Custom IRQ priorities

[JamesHarris]

You guys might find this of interest. I didn't think it was possible to apply custom, arbitrary prioritisation to IRQs - at least with the traditional 8259 PICs - but there may be a way.

Here's the proposal. Feel free to comment.

The idea is to get the PICs to inform us immediately of any IRQs which are requesting service so we gain a complete picture of all IRQs which require attention. Then, since we will know of all IRQs which need attention we can process them in whatever order we want.

The key change from normal is for an interrupt to be EOId before it is handled rather than afterwards, and for the relevant IMR to be used to prevent that same IRQ from firing again.

As well as masking an IRQ it would be added to a waiting set. The waiting set could be as simple as a bit array with one bit for each IRQ number or it could be more extensive but the point is that it would keep track of which IRQs had been signalled but had not yet been processed.

To illustrate, here's a piece of pseudocode to show what would happen when a hardware interrupt comes in. Before it starts EOI would have been issued automatically by AEOI (on all but very old hardware).

On an interrupt:

Code: Select all

  Push registers
  Mask this interrupt in the relevant IMR
  Add this interrupt to the set of waiting interrupts
  ... body ...
  Pop registers
  iret

In other words, aside from saving and restoring registers all that would be done would be to mask off the current IRQ and to leave a note that the IRQ is awaiting service.

For example, say half a dozen interrupts all fire at once. Each of the six would, in turn, follow the above code path. That would leave all six masked off and recorded as awaiting service.

Naturally, something has to do the actual servicing. AISI it could be one of the interrupts or a high-priority task.

In the pseudocode below I'll make it one of the interrupts. I'll use a nesting level to determine whether an interrupt is already being serviced or not. Nesting level will be zero on initial entry and will be 1 if one or more interrupts are being serviced. There would only be the two levels.

The code in the above which is shown as "... body ..." would be

Code: Select all

  If nesting level is zero
    Increment nesting level
    Loop while there are any interrupts waiting
      Pick the IRQ we regard as of highest priority
      Delete it from waiting list
      Enable interrupts
      Handle the IRQ
      Disable interrupts
      Unmask the IRQ
    Endloop
    Decrement nesting level
  Endif

The loop would be able to take interrupts in any order we wanted, thus implementing custom prioritisation, and it would either unmask each one as it completed it or unmask them all at the end (or some combination thereof).

If any new interrupts fired while the code was running they would be added to the waiting set and then would be processed in their turn before the loop completed.

Only once the waiting set was empty would the loop terminate. The code would then, as normal, ireq back to whatever had been interrupted.

I think it would work but it is at present untested. I have in mind some potential caveats and a number of potential improvements but I think the above is enough to show the basic idea.

FWIW I'd probably give the IRQs the following priorities:

• 0 PIT
  8 RTC
  3 Serial
  4 Serial
  all the rest ...

In other words, I'd want to prioritise serial ports while ensuring that correct time is kept.

[nullplan]

So, the interaction with the PIC here is the same as what would take place in the threaded interrupt handlers Linux has (which I suggested to you previously). The only difference is you are handling all interrupts on the same stack. Not sure if that's an issue, but it might be later on. You have to condition the enabling of interrupts on there being "enough" stack left to do so, for some measure of "enough".

I seriously doubt interrupt priority is all that important, but I lack the data to back that up. Reason I think this is that modern computers are so fast, interrupts should be handled so quickly, that it rarely happens that multiple interrupts are firing at the same time. And even then, the handlers are done so quickly it makes no measurable difference which one is taken first.

In your scheme, interrupts are still largely taken in the order they arrive in. Only if multiple interrupts arrive while interrupts are disabled in the CPU will you actually get any choice in the matter. And, of course, now your interrupt handlers are being interrupted. I'm sure your RTC appreciates being left waiting while you're tending to the serial ports that just popped up, The only way to avoid that would be to mask out all lower-priority interrupts as you are serving an interrupt, but that means that instead of just masking out all interrupts, which is possible with a single instruction and indeed happens automatically, you now have to reconfigure the IMR constantly, and in modern PCs, the PIC is very slow to talk to.

Also, it creates two classes of code: Interrupt code and non-interrupt code, and the non-interrupt code can go starve for all your scheme cares about. There is no way to interleave non-interrupt code in between the interrupt handlers. If, after reading a network packet from the Ethernet card, you see that the PIT wants attention, but you really need to get a move on and let that batch processing job have a go for a bit, you have no way to do so. The threaded interrupt handler model solves this by making all interrupt handlers go through the normal scheduler, and so nothing can starve (unless your scheduler is bad, but at least that is a solvable problem).

Oh, and for time keeping: Just use hardware counters. It's what I do. Depending on system, use the TSC, the HPET, or some other counter that is present almost everywhere (can't you even read out the PIT counters?), and then when someone wants to know the time, you just ask the hardware counter how far it counted. Take the difference from some known value in the past, normalize the result, and done. No need to interrupt software regularly to count interrupts, when you can just ask hardware when you need the information.

[JamesHarris]

In your scheme, interrupts are still largely taken in the order they arrive in. Only if multiple interrupts arrive while interrupts are disabled in the CPU will you actually get any choice in the matter.

Not exactly. If interrupts occur individually then, yes, they would be processed in the order in which they occur but if, say, ten IRQs were to fire concurrently then there would be an initial acceptance loop which records all ten one at a time but then all ten would be serviced in whatever order is desired.

More importantly, however, if a high-priority interrupt were to fire while the ten were being processed then it would be handled before any remaining lower-priority ones.

And, of course, now your interrupt handlers are being interrupted.

CLI/STI could prevent interruption where necessary.

Please note that in the algorithm presented they could be interrupted but not preempted. Any IRQ which fired while a service routine was running would do no more than note that the IRQ had fired and then return to the service routine.

But the proposal is also adaptable to providing preemption, if desired.

I'm sure your RTC appreciates being left waiting while you're tending to the serial ports that just popped up,

No need. CLI/STI would bracket the RTC access.

Will reply to the rest of your points separately.

[JamesHarris]

... you now have to reconfigure the IMR constantly, and in modern PCs, the PIC is very slow to talk to.

That's not so. The IMR could be written to just once or twice per interrupt depending on what was preferred but that's not going to vary much in cost from the EOI or two which are typically sent in standard servicing and would not be needed here.

Also, it creates two classes of code: Interrupt code and non-interrupt code,

True. As is fairly normal, I would suggest.

... and the non-interrupt code can go starve for all your scheme cares about.

It would have overcomplicated the example to have included code to handle it but it is something I've aware of. I have in mind what you could call 'limp home mode'! which would be along the lines that you suggest. Unlike your suggestion, however, it would be only for specific IRQs which are put in 'special measures' because they won't shut up. For something like that your suggestion would be a boon.

There is no way to interleave non-interrupt code in between the interrupt handlers.

Correct.

[JamesHarris]

If, after reading a network packet from the Ethernet card, you see that the PIT wants attention, but you really need to get a move on and let that batch processing job have a go for a bit, you have no way to do so.

Well, I don't think the PIT itself would need any attention. All that would happen in its interrupt service routine is that a counter or two would be updated in memory and possibly a flag set to indicate that a task switch should be considered on return from interrupt mode to task mode. It would be insane to try to run part of a batch job rather than doing such a short time-sensitive update.

The threaded interrupt handler model solves this by making all interrupt handlers go through the normal scheduler, and so nothing can starve (unless your scheduler is bad, but at least that is a solvable problem).

I understand that but have you actually done any calculations of the relative latencies of using the general scheduler to schedule interrupts with that of minimal handling in interrupt time?

By minimal I mean clearing the cause of the IRQ and leaving anything else to be handled by a generally schedulable task.

Oh, and for time keeping: Just use hardware counters.

Rest assured that on machines which have em I'll use em! The PIT is for fallback.

No need to interrupt software regularly to count interrupts, when you can just ask hardware when you need the information.

Good point.