8259 PIC special mask mode

[JamesHarris]

Is there any way with 8259s to implement a custom interrupt priority order?

To illustrate what I mean, one could consider the PC/AT's normal, standard priority order to be

0 - PIT
1 - Keyboard
8...15 - RTC and rest of slave PIC
3 - Serial port
4 - Serial port
5...7 - Rest of master PIC

I'd prefer something like

0 - PIT
8 - RTC
3 - Serial port
4 - Serial port
Everything else

In other words, timers then serial ports then the rest.

You may prefer a different order but is there any way to achieve it?

That's where I'm wondering if special mask mode might help. I've read the description of it in the datasheet more than once before but never understood it. I still don't even now. :-(

But on reading it again it sounds as though it /might/ be usable for such custom prioritisation.

The Intel 8259A datasheet says this about it:

"Special Mask Mode

"Some applications may require an interrupt service routine to dynamically alter the system priority structure during its execution under software control. For example, the routine may wish to inhibit lower priority requests for a portion of its execution but enable some of them for another portion.

"The difficulty here is that if an Interrupt Request is acknowledged and an End of Interrupt command did not reset its IS bit (i.e., while executing a service routine), the 8259A would have inhibited all lower priority requests with no easy way for the routine to enable them.

"That is where the Special Mask Mode comes in. In the special Mask Mode, when a mask bit is set in OCW1, it inhibits further interrupts at that level and enables interrupts from all other levels (lower as well as higher) that are not masked. Thus, any interrupts may be selectively enabled by loading the mask register."

See what I mean? It sounds as thought special mask mode might allow custom priorities (and more) through judicious masking of interrupts.

How does it look to you and do you know of any other way to effect custom interrupt priorities such as those I mentioned at the outset?

[kzinti]

Two things come to mind:

1) If your interrupt handlers simply post a message to a thread, you can set the priority of each interrupt handling thread to whatever you want. This way priority is set using the scheduler. For example, when a key is pressed, the interrupt handler would send a message to a keyboard thread (the message is just a signal, you don't need anything fancy here). The keyboard thread then does the handling. Same with the timer and other interrupts. All these interrupt handling threads would then get scheduler based on their priority. I remember Brendan talking about something like this in the past and this makes a lot of sense to me.

I think the "Special Mask Mode" you are describing is there to help implement something like this. That's my 2 min read anyways.

2) I think that once you start using APICs, you are not going to care much about the 8259 PIC anymore

[JamesHarris]

Two things come to mind:

1) If your interrupt handlers simply post a message to a thread, you can set the priority of each interrupt handling thread to whatever you want. This way priority is set using the scheduler. For example, when a key is pressed, the interrupt handler would send a message to a keyboard thread (the message is just a signal, you don't need anything fancy here). The keyboard thread then does the handling. Same with the timer and other interrupts. All these interrupt handling threads would then get scheduler based on their priority. I remember Brendan talking about something like this in the past and this makes a lot of sense to me.

I may do that for a number of interrupt responses but, even then, in the interrupt handlers I still need to service devices at least as far as getting them to deassert their interrupt requests.

As for timings and why this is important, humble things like serial ports can be incredibly demanding. AISI four serial ports at 57,600 baud could generate over 40,000 interrupts per second! That means they would have to be serviced as a priority, especially on older hardware in which the UARTs have no FIFOs and bytes have to be handled one at a time. So while I've not timed real code for this I suspect that passing messages to and from a task would add too much overhead.

I am not sure even about getting that many events per second to be handled successfully just as interrupts but ISTM best to keep interrupt latencies low and to handle IRQ priorities as mentioned in this thread.

I think the "Special Mask Mode" you are describing is there to help implement something like this. That's my 2 min read anyways.

Cool.

2) I think that once you start using APICs, you are not going to care much about the 8259 PIC anymore

Point taken. However, I'm after the compatibility of supporting both old and new hardware so still would need to work with the 8259 PICs.

[nullplan]

I may do that for a number of interrupt responses but, even then, in the interrupt handlers I still need to service devices at least as far as getting them to deassert their interrupt requests.

No you don't. You mask out the interrupt line in the interrupt handler itself. Then, once the actual handler has run, and has determined that you want to get the interrupt again, you unmask the interrupt. Pretty straightforward.

AISI four serial ports at 57,600 baud could generate over 40,000 interrupts per second!

Let's assume 8/0/1 settings on the terminal (8 data bits, no parity, 1 stop bit). Plus the start bit, that makes 10 bits per byte. Baudrate is roughly equal to bit rate, so at 57600 baud, you can receive up to 5760 characters per second. Assuming you receive and transmit at the same time, you still only get twice that in interrupts, or roughly 10,000 interrupts. No idea where you got the 40,000 from.

So while I've not timed real code for this I suspect that passing messages to and from a task would add too much overhead.

Never presume overhead. Build a system that works, measure its performance. Come back once you see it is too slow.

[rdos]

I may do that for a number of interrupt responses but, even then, in the interrupt handlers I still need to service devices at least as far as getting them to deassert their interrupt requests.

No you don't. You mask out the interrupt line in the interrupt handler itself. Then, once the actual handler has run, and has determined that you want to get the interrupt again, you unmask the interrupt. Pretty straightforward.

I don't think that is a good idea. I think you should handle the cause of the interrupt in the interrupt handler, but then you can defer the actions needed to be done in the OS itself to a server thread.

AISI four serial ports at 57,600 baud could generate over 40,000 interrupts per second!

Let's assume 8/0/1 settings on the terminal (8 data bits, no parity, 1 stop bit). Plus the start bit, that makes 10 bits per byte. Baudrate is roughly equal to bit rate, so at 57600 baud, you can receive up to 5760 characters per second. Assuming you receive and transmit at the same time, you still only get twice that in interrupts, or roughly 10,000 interrupts. No idea where you got the 40,000 from.

Still, serial interrupts are a real challenge. A serial port can run up to 115,000 baud, which might be where the figure 40,000 comes from. While I have server threads for most devices, I handle the serial port in the IRQ. It's mostly as simple as using a queue protected with a spinlock where you save incoming characters and fetch outgoing. There is no sense in off-loading this to a server thread.

[JamesHarris]

I may do that for a number of interrupt responses but, even then, in the interrupt handlers I still need to service devices at least as far as getting them to deassert their interrupt requests.

No you don't. You mask out the interrupt line in the interrupt handler itself. Then, once the actual handler has run, and has determined that you want to get the interrupt again, you unmask the interrupt. Pretty straightforward.

Interesting idea. I have my doubts about the latency it would introduce but I'll keep it in mind.

AISI four serial ports at 57,600 baud could generate over 40,000 interrupts per second!

Let's assume 8/0/1 settings on the terminal (8 data bits, no parity, 1 stop bit). Plus the start bit, that makes 10 bits per byte. Baudrate is roughly equal to bit rate, so at 57600 baud, you can receive up to 5760 characters per second. Assuming you receive and transmit at the same time, you still only get twice that in interrupts, or roughly 10,000 interrupts. No idea where you got the 40,000 from.

Four ports.

So while I've not timed real code for this I suspect that passing messages to and from a task would add too much overhead.

Never presume overhead. Build a system that works, measure its performance. Come back once you see it is too slow.

In my time I've come across lots of software which is orders of magnitude slower than it should be because it is riddled with all kinds of inefficiencies (which are then executed in loops, compounding the problems) and ISTM better to design for performance from the outset.

That's especially important when it comes to (1) interrupt latency and (2) the fact that an OS cannot be tested on all machines it may have to run on.

Besides, it looks as though even on the old PICs it will be possible to get full scheduling control over the order in which interrupts are handled. So why not do it?!

[JamesHarris]

Still, serial interrupts are a real challenge.

Indeed. It is somewhat counterintuitive but one of the slowest devices can pose one of the greatest challenges to the hardware and the OS software.

To put it in context, though, bidirectional Gigabit ethernet can have, say, 200,000 packets per second. Yet AIUI they will at least be batched up - as will serial port IO in all but very old machines - and at least gigiabit NICs will only be found in more-capable machines than some of those which have async serial ports.

A serial port can run up to 115,000 baud, which might be where the figure 40,000 comes from.

The 40,000 comes from four ports but I was being reasonably conservative. I believe 16550 UARTs can run at twice the speed and their early models did not have a working FIFO. See https://en.wikipedia.org/wiki/16550_UART#The_16550_FIFO for details.

While I have server threads for most devices, I handle the serial port in the IRQ. It's mostly as simple as using a queue protected with a spinlock where you save incoming characters and fetch outgoing. There is no sense in off-loading this to a server thread.

A spinlock in an interrupt service routine?

[rdos]

To put it in context, though, bidirectional Gigabit ethernet can have, say, 200,000 packets per second. Yet AIUI they will at least be batched up - as will serial port IO in all but very old machines - and at least gigiabit NICs will only be found in more-capable machines than some of those which have async serial ports.

NICs typically have schedules that buffer many packets in memory. Therefore, the server thread approach works for NICs. For serial ports, you must handle the current character before the next one is ready, otherwise, you will miss characters, and so it's easy to miss characters if you have too long interrupt latency.

A spinlock in an interrupt service routine?

Yes, you must have some ordinary code that also must manipulate the serial queues, and accessing those will need spinlocks on multicore. On single-core, just disabling interrupts will do, but not on multicore.

[kzinti]

I suspect that passing messages to and from a task would add too much overhead.

You just need 1 bit of signal per IRQ. There is no need for any heavy message passing here.

Sending a message can just mean releasing a semaphore.

For each IRQ, you would have one semaphore and a thread blocked on it. When the IRQ fires, you just signal the thread using the semaphore. When returning from the IRQ handler, you check if the scheduler should switch to that thread that was just released.

I can't imagine there is a cheaper way to make this work if you really want to define IRQ priorities yourself and not be dependent on how the PIC is wired.

[Ethin]

I may do that for a number of interrupt responses but, even then, in the interrupt handlers I still need to service devices at least as far as getting them to deassert their interrupt requests.

No you don't. You mask out the interrupt line in the interrupt handler itself. Then, once the actual handler has run, and has determined that you want to get the interrupt again, you unmask the interrupt. Pretty straightforward.

Interesting idea. I have my doubts about the latency it would introduce but I'll keep it in mind.

AISI four serial ports at 57,600 baud could generate over 40,000 interrupts per second!

Let's assume 8/0/1 settings on the terminal (8 data bits, no parity, 1 stop bit). Plus the start bit, that makes 10 bits per byte. Baudrate is roughly equal to bit rate, so at 57600 baud, you can receive up to 5760 characters per second. Assuming you receive and transmit at the same time, you still only get twice that in interrupts, or roughly 10,000 interrupts. No idea where you got the 40,000 from.

Four ports.

So while I've not timed real code for this I suspect that passing messages to and from a task would add too much overhead.

Never presume overhead. Build a system that works, measure its performance. Come back once you see it is too slow.

In my time I've come across lots of software which is orders of magnitude slower than it should be because it is riddled with all kinds of inefficiencies (which are then executed in loops, compounding the problems) and ISTM better to design for performance from the outset.

That's especially important when it comes to (1) interrupt latency and (2) the fact that an OS cannot be tested on all machines it may have to run on.

Besides, it looks as though even on the old PICs it will be possible to get full scheduling control over the order in which interrupts are handled. So why not do it?!

Is the software you've come across that's inefficient an operating system? If not, then you can't really apply that kind of measurement here. An operating system is a completely different beast to a shell that receives signals from the kernel and reacts slowly, for example.
Unless you can prove that the PIC is slow, design your ISR to immediately read the byte and then process it like you normally would. The very optimization attempts you are trying here could easily make your code a lot more inefficient than processing it without them. Write your ISR to process the interrupt and how fast it is.
About your point on performance, I very much disagree. Performance is important, true, but so is productivity and the ability to understand the code. Focus on a good design for your system before you go hand-optimizing things. And, as I've said before, profile your ISR. Set your baud rate to 115200. If your ISR doesn't respond fast enough when you spam it with thousands of characters, then optimize it, but don't do so without knowing that its necessary first. As I said, your handcrafted optimizations might just have the opposite effect of what your aiming for.

[nullplan]

Interesting idea. I have my doubts about the latency it would introduce but I'll keep it in mind.

I don't know about latency, but I can vouch for its stability. This is, in essence, what Linux is doing. I once had a system set up wrong, an interrupt trigger was set to level on a GPIO pin, and that pin was always at that level. So the system was in an interrupt storm, constantly being interrupted. And you almost wouldn't notice. Most things still worked normally. Some timing things were off, but others worked perfectly.

A spinlock in an interrupt service routine?

Not a problem if spinlocks disable interrupts. Spinlocks are not meant to be held for a long time. The holder of a spinlock must not sleep, or go to userspace, and so it will release the spinlock in a very short amount of time. This is the difference between a spinlock and a mutex. And you are right, you must never take a mutex lock in an interrupt handler.

In my time I've come across lots of software which is orders of magnitude slower than it should be because it is riddled with all kinds of inefficiencies (which are then executed in loops, compounding the problems) and ISTM better to design for performance from the outset.

But you can only do that once you have understood the problem. You needed to build the inefficient but working system first to see where the inefficiencies really mount up, and to try and think of a better way. Yes, sometimes you can have great fun doing local optimizations on an inefficient design, hammering down the nails that stick out, and other times no nails really stick out but things aren't all that great, either. However, it is still better to first build a system that works and then improve on the design.

Also, if you have no hot code, it is possible you just have hot data. In which case an analysis of the data flow (and an optimization there) might be in order. You still never preemptively optimize, but rather, measure the impact of your design changes. Programming can be very counter-intuitive.

I don't think that is a good idea. I think you should handle the cause of the interrupt in the interrupt handler, but then you can defer the actions needed to be done in the OS itself to a server thread.

This is consistent with previous statements from you (which boil down to "Linux is wrong" in so many words[1]), yet this does not address the actual problem. The problem is that interrupts arrive when they damn well please, and the hardware has fixed the interrupt priorities, and there is nothing the software can do about it. And that is still true with the proposed masking scheme, but the actual interrupt handlers themselves are minimized, and the approach is portable to all other interrupt controllers, and generic to all devices. Your approach would have us calling driver code from the ISR, which is what I was trying to avoid. The deferment proposed by me allows the "ISR" to run in kernel thread context, and do way more things than a real ISR could.

[1]I do not know if this is conscious on your part, but you seem to be going the other direction whenever I present anything from Linux's book here. Especially the less intuitive parts.

[rdos]

]This is consistent with previous statements from you (which boil down to "Linux is wrong" in so many words[1]), yet this does not address the actual problem. The problem is that interrupts arrive when they damn well please, and the hardware has fixed the interrupt priorities, and there is nothing the software can do about it. And that is still true with the proposed masking scheme, but the actual interrupt handlers themselves are minimized, and the approach is portable to all other interrupt controllers, and generic to all devices. Your approach would have us calling driver code from the ISR, which is what I was trying to avoid. The deferment proposed by me allows the "ISR" to run in kernel thread context, and do way more things than a real ISR could.

[1]I do not know if this is conscious on your part, but you seem to be going the other direction whenever I present anything from Linux's book here. Especially the less intuitive parts.

As far as I understand both the Windows and Linux interrupt system design it's a layered design where extremely complex code can run at different "interrupt" levels. I find this design highly unwanted, and impossible to debug. My design builds on ALWAYS linking the IRQ to the driver code, and then the driver IRQ will either defer further processing to a server thread (by signaling it) or do some of its own processing. Most devices will only remove the interrupt condition and then signal the server thread, while the serial port IRQ will read out characters (or send characters) in the IRQ and use a spinlock to synchronize the queue with the driver.

It's easy to debug server threads running in kernel space without intrusive measures while debugging IRQ-level code requires creating panics and going into a monitor that can only be excited by a reboot. Using the Windows design would make it necessary to debug most interrupt activity at the panic level.

Edit: By linking the IRQ I mean letting the driver register a callback that the interrupt controller will call when the interrupt happens, not linking it directly. All IRQs have a common part, which among other things will wake up server threads signaled in the driver part. The common part will also disable the scheduler and enable interrupts to improve interrupt latency. The interrupt controller (PIC or APIC) will also insert EOI and other required parts. Some IRQs can be shared (for instance PCI IRQs), and so there is support for linking multiple driver callbacks. Actually, the interrupt controller will create dynamic IRQ stubs on the fly that do the callbacks and thus is not based on some common code. Linking more handlers can be done on the fly too since each part of the chain has a next pointer that can be modified.

[JamesHarris]

I once had a system set up wrong, an interrupt trigger was set to level on a GPIO pin, and that pin was always at that level. So the system was in an interrupt storm, constantly being interrupted. And you almost wouldn't notice. Most things still worked normally. Some timing things were off, but others worked perfectly.

That's surprising! I would have thought the CPU would have never got back out of interrupt mode. Did you work out how it was that the CPU got any work done?

That said, the interrupt ack process may be quite slow. Could it have been that the CPU got a tiny amount of work done (especially that which did not involve much bus time) in the period between the iret reenabling interrupts and it accepting the new interruption - e.g. while the CPU was communicating with the PICs?

You still never preemptively optimize, but rather, measure the impact of your design changes.

Indeed. I am not talking about optimisation but design. AISI it's normal to design for performance and to optimise later once one has a working system.

[Korona]

If we take the 40k IRQs per second for granted (which is already insane) and you assume that your code runs on an ancient i386 with 40 MHz, you still have 1000 instructions to handle one FIFO IRQ, i.e., more than enough to read from the data port and wake up a thread, even without dealing with nested interrupts and IRQ priorities at all.

I second nullplan's remarks about actually building a system and measuring -- the claim that you need interrupt priorities at all to keep latency down is not supported by any data at this point.

[nullplan]

That's surprising! I would have thought the CPU would have never got back out of interrupt mode. Did you work out how it was that the CPU got any work done?

As I said, interrupts were masked while being handled, and the handling was concurrent with the other processing. The GPIO interrupt would occur, then the GPIO interrupt was masked out, and the CPU would return to normal work. At some point, the GPIO interrupt thread would run, causing a cascade of virtual interrupts to be triggered, and only when that was all done would the GPIO interrupt be re-enabled. And in all of that time, timer interrupts could still work, and multitasking was continuing as normal.

That said, the interrupt ack process may be quite slow. Could it have been that the CPU got a tiny amount of work done (especially that which did not involve much bus time) in the period between the iret reenabling interrupts and it accepting the new interruption - e.g. while the CPU was communicating with the PICs?

Well, it was an ARM system, so no IRET. But the CPU could just work normally between masking out the GPIO interrupt and re-enabling it.

AISI it's normal to design for performance and to optimise later once one has a working system.

I've heard of designing for performance. I think that is a bad idea. You want to design for ease of understanding, for readable and testable code. There are more important things to consider than performance at the outset, especially when you haven't yet quite understood the problem. Build a system that is understandable, readable, and testable. And then, if performance is indeed lacking, address the shortcomings.

"Performance first" sounds good, but leaves too many other things at the wayside.

If we take the 40k IRQs per second for granted (which is already insane) and you assume that your code runs on an ancient i386 with 40 MHz, you still have 1000 instructions to handle one FIFO IRQ, i.e., more than enough to read from the data port and wake up a thread, even without dealing with nested interrupts and IRQ priorities at all.

1000 ticks != 1000 instructions. Especially if there is something as costly as an interrupt transition in there. All that IDT dereferencing and frame pushing doesn't come for free.