Context switch on timer interrupt

[bzt]

Wrt to IST, you only support long mode?

Yes, on x86_64 only long mode supported. I'm only interested in 64 bit, and I've waited long for a RPi3 with amrv8.

bzt, what happens to your code if another IRQ occurs before your "sub/add pair" and the same IST entry is used, thus the new IRQ will overwrite the stack before you saved it? Also why do you use "cli" as your first instruction in the ISR?

It seems you have already answered your question The cli makes sure of it that no other interrupt will be fired before I modify the IST. I'd like to point out that originally I've planned to issue an sti before I call the C code and a cli afterwards, but since my IRQ handler does nothing more than sending a message to a task (which is atomic and cannot be interrupted), it turned out to be more efficient just leave it as is. Exceptions are a different case, there I'll have to enable interrupts for sure, as they do more time consuming tasks than storing 64 bytes.

[iansjack]

I'm with you. It only makes sense to target x86_64 nowadays. 32-bit is just too limited.

[LtG]

Wrt to IST, you only support long mode?

Yes, on x86_64 only long mode supported. I'm only interested in 64 bit, and I've waited long for a RPi3 with amrv8.

bzt, what happens to your code if another IRQ occurs before your "sub/add pair" and the same IST entry is used, thus the new IRQ will overwrite the stack before you saved it? Also why do you use "cli" as your first instruction in the ISR?

It seems you have already answered your question The cli makes sure of it that no other interrupt will be fired before I modify the IST. I'd like to point out that originally I've planned to issue an sti before I call the C code and a cli afterwards, but since my IRQ handler does nothing more than sending a message to a task (which is atomic and cannot be interrupted), it turned out to be more efficient just leave it as is. Exceptions are a different case, there I'll have to enable interrupts for sure, as they do more time consuming tasks than storing 64 bytes.

I didn't answer my own question, the reason I asked about CLI was because assuming you use interrupt gate then interrupts are already disabled, the CLI is redundant and serves no purpose.

My point was that if the same IST entry is used for example for an NMI then the NMI could occur before your "sub/add pair", because NMI's aren't prevented by CLI.

[LtG]

I'm with you. It only makes sense to target x86_64 nowadays. 32-bit is just too limited.

I don't think 32-bit is that limited, but due to time constraints I've also decided to focus only on 64-bit and eventually (possibly) tackle 32-bit if 32-bit is even remotely relevant by the time I get there, which it realistically might not be =)

[bzt]

I didn't answer my own question, the reason I asked about CLI was because assuming you use interrupt gate then interrupts are already disabled, the CLI is redundant and serves no purpose.

My point was that if the same IST entry is used for example for an NMI then the NMI could occur before your "sub/add pair", because NMI's aren't prevented by CLI.

Well, I've made some tests, and you are right, if the ISR is triggered by an IRQ, IF is already cleared. But not in case of software interrupt. So I won't remove cli because I like failsafes Now let's examine each option here one by one:
1. software interrupt: cannot happen as there's no int instruction in my ISR
2. hardware interrupt: cannot happen either, as IRQs are masked (no EOI issued and IF cleared)
3. NMI: it can happen, as it's not masked, but it uses a different IST so no problem in this case either.
Happening another interrupt during NMI: software interrupts out of question. IRQs are still masked. I'm not sure whether NMI could happen during NMI ISR (is it masked automatically or needs explicit masking?), but if it can, it's so very unlikely, so rare case which I don't care about. Frankly I don't care much about NMI as it will be triggered only in case of some serious, non-recoverable hardware error; and if the memory is faulty, I cannot guarantee that my OS won't crash anyway.

[LtG]

Well, I've made some tests, and you are right, if the ISR is triggered by an IRQ, IF is already cleared. But not in case of software interrupt. So I won't remove cli because I like failsafes Now let's examine each option here one by one:

Why not add "CLI" three times, just to be "safe"?

Note, interrupt gate disable interrupts (by clearing IF in EFLAGS, CLI works by clearing IF in EFLAGS, see, exactly the same), so how is manually disabling interrupts "safer"? A trap gate does not disable interrupts and that's the difference between trap and interrupt gates.

[bzt]

Well, I've made some tests, and you are right, if the ISR is triggered by an IRQ, IF is already cleared. But not in case of software interrupt. So I won't remove cli because I like failsafes Now let's examine each option here one by one:

Why not add "CLI" three times, just to be "safe"?

Note, interrupt gate disable interrupts (by clearing IF in EFLAGS, CLI works by clearing IF in EFLAGS, see, exactly the same), so how is manually disabling interrupts "safer"? A trap gate does not disable interrupts and that's the difference between trap and interrupt gates.

Wrong. An interrupt gate disables interrupts only and only if triggered by hardware. Read spec carefully or do some tests if you don't believe. As I've already said, software interrupts does not clear IF. So ONE cli as the first instruction is required to be bullet-proof (If there is a way to call ISR with int 0xXX, my ISR must handle that case correctly too. That single cli command won't cause much overhead). 3 clis are simply stupid and pointless.

[Brendan]

Hi,

Well, I've made some tests, and you are right, if the ISR is triggered by an IRQ, IF is already cleared. But not in case of software interrupt. So I won't remove cli because I like failsafes Now let's examine each option here one by one:

Why not add "CLI" three times, just to be "safe"?

Note, interrupt gate disable interrupts (by clearing IF in EFLAGS, CLI works by clearing IF in EFLAGS, see, exactly the same), so how is manually disabling interrupts "safer"? A trap gate does not disable interrupts and that's the difference between trap and interrupt gates.

Wrong. An interrupt gate disables interrupts only and only if triggered by hardware. Read spec carefully or do some tests if you don't believe. As I've already said, software interrupts does not clear IF. So ONE cli as the first instruction is required to be bullet-proof (If there is a way to call ISR with int 0xXX, my ISR must handle that case correctly too. That single cli command won't cause much overhead). 3 clis are simply stupid and pointless.

No, this is very wrong. For interrupt gates the CPU clears IF, regardless of whether the interrupt was started by an exception, IRQ, software interrupt or anything else.

The only potential case where this might or might not be true is if the interrupt handler runs at "CPL > IOPL" (the interrupt handler doesn't have high enough privilege to touch IF); which is something that almost never makes any sense, and something where "CLI" wouldn't be permitted either.


Cheers,

Brendan

[bzt]

No, this is very wrong. For interrupt gates the CPU clears IF, regardless of whether the interrupt was started by an exception, IRQ, software interrupt or anything else.

The only potential case where this might or might not be true is if the interrupt handler runs at "CPL > IOPL" (the interrupt handler doesn't have high enough privilege to touch IF); which is something that almost never makes any sense, and something where "CLI" wouldn't be permitted either.


Cheers,

Brendan

Hmmm, interesting. I can confirm that my emulator does not clear IF bit on soft interrupts, I've tested it twice. Maybe a bug? I'll update it. You are right according to the spec IF should be cleared, regardless of what triggered the gate.
It also states that NMI is automatically masked in an NMI handler, so one should not worry about that case.

[LtG]

Hmmm, interesting. I can confirm that my emulator does not clear IF bit on soft interrupts, I've tested it twice. Maybe a bug? I'll update it. You are right according to the spec IF should be cleared, regardless of what triggered the gate.
It also states that NMI is automatically masked in an NMI handler, so one should not worry about that case.

Which emulator? And let us know how updating affects it.. How did you test it?

As for NMI, there's a thread about that http://forum.osdev.org/viewtopic.php?f=1&t=31486, basically the main issue is that if an SMI occurs during your NMI then the SMI handler (in firmware, which you can't control) _may_ re-enable NMI's, and if it does that then it's possible for NMI's to nest. But if you're not aiming at very high stability then it's probably something that won't matter, besides if you do get an NMI there's probably not much you can do besides kpanic..

[bzt]

Which emulator? And let us know how updating affects it.. How did you test it?

One that's source has been messed with way too much. Vanilla bochs works. How did I test it? Are you serious? Imagine, I've used debugger prompt and examined registers, namely rflags... Don't try it, it's magic! It's very dangerous in uninitiated hands!

As for NMI, there's a thread about that http://forum.osdev.org/viewtopic.php?f=1&t=31486, basically the main issue is that if an SMI occurs during your NMI then the SMI handler (in firmware, which you can't control) _may_ re-enable NMI's, and if it does that then it's possible for NMI's to nest. But if you're not aiming at very high stability then it's probably something that won't matter, besides if you do get an NMI there's probably not much you can do besides kpanic..

I don't understand you, and I mean it. What exactly do you expect to happen, if the hardware reports unrecoverable hardware failure? You think that your NMI handler would somehow fix the underlying hardware and continue normal operation without problem, but that nasty SMI would prevent that to happen?
If there's not much you can do besides displaying a kpanic, then how would adding more complex code improve stability? Stability in what?

[LtG]

As for NMI, there's a thread about that http://forum.osdev.org/viewtopic.php?f=1&t=31486, basically the main issue is that if an SMI occurs during your NMI then the SMI handler (in firmware, which you can't control) _may_ re-enable NMI's, and if it does that then it's possible for NMI's to nest. But if you're not aiming at very high stability then it's probably something that won't matter, besides if you do get an NMI there's probably not much you can do besides kpanic..

I don't understand you, and I mean it. What exactly do you expect to happen, if the hardware reports unrecoverable hardware failure? You think that your NMI handler would somehow fix the underlying hardware and continue normal operation without problem, but that nasty SMI would prevent that to happen?
If there's not much you can do besides displaying a kpanic, then how would adding more complex code improve stability? Stability in what?

Not sure which part you didn't understand..?

I was saying that if you aren't planning on attempting to recover from NMI then the NMI-SMI-NMI is largely/completely irrelevant for you. As such doing kpanic might be a good idea, which you seem to have stated as well.

As for possibly handling NMI's, that largely depends on what caused the NMI. I'm not sure what causes modern hardware to issue NMI, but for instance if the NMI was caused by memory corruption then certainly there's plenty that can be done about it. If it's caused by something that is by definition "unrecoverable", then of course it's unrecoverable.

One other thing about NMI's and attempting to handle them is the difficulty of trying to test it in practice, which might also be enough of a reason to not even attempt it.

[Brendan]

Hi,

As for NMI, there's a thread about that http://forum.osdev.org/viewtopic.php?f=1&t=31486, basically the main issue is that if an SMI occurs during your NMI then the SMI handler (in firmware, which you can't control) _may_ re-enable NMI's, and if it does that then it's possible for NMI's to nest. But if you're not aiming at very high stability then it's probably something that won't matter, besides if you do get an NMI there's probably not much you can do besides kpanic..

I don't understand you, and I mean it. What exactly do you expect to happen, if the hardware reports unrecoverable hardware failure? You think that your NMI handler would somehow fix the underlying hardware and continue normal operation without problem, but that nasty SMI would prevent that to happen?
If there's not much you can do besides displaying a kpanic, then how would adding more complex code improve stability? Stability in what?

Not sure which part you didn't understand..?

I was saying that if you aren't planning on attempting to recover from NMI then the NMI-SMI-NMI is largely/completely irrelevant for you. As such doing kpanic might be a good idea, which you seem to have stated as well.

Even for kpanic; the goal is to continue for long enough to provide the user with relevant information (e.g. the fact that the panic was caused by an NMI and some details of what the CPU was doing when it happened) and stop other CPUs from running (and making a mess of things); and crashing (e.g. due to NMI-SMI-NMI) before you've done these things would be undesirable.

As for possibly handling NMI's, that largely depends on what caused the NMI. I'm not sure what causes modern hardware to issue NMI, but for instance if the NMI was caused by memory corruption then certainly there's plenty that can be done about it. If it's caused by something that is by definition "unrecoverable", then of course it's unrecoverable.

One other thing about NMI's and attempting to handle them is the difficulty of trying to test it in practice, which might also be enough of a reason to not even attempt it.

Some servers come with an "NMI button"; so that if an administrator notices an OS has locked up (e.g. maybe something like a spinlock that's spinning forever with IRQs disabled) they can press the button to get some idea of where the problem is (or get some information that makes a bug report better than useless). The other common cause is watchdog timers; which is basically an automated "tell me what the OS was doing if it locks up" alternative to the NMI button (e.g. normally the OS updates a timer regularly to stop it from expiring; and when the OS locks up it doesn't update the timer, so the timer expires and sends NMI).

Note: For hardware errors (e.g. memory corruption), I think most of it has been shifted to the machine check system in modern 80x86 computers. Unfortunately, if you go digging in modern chipset datasheets you'll still find (e.g.) registers that control things like "if FOO happens; send NMI or #SERR or SCI" and won't have any idea how the firmware actually configured the chipset (and can't easily create a list of things that cause NMI for a specific chipset).

For testing; one thing OS developers could really benefit from is an emulator designed for fault emulation and fault injection. Even basic things (e.g. checking if your "software RAID" layer actually does recover from hard drive failures properly) are excessively hard to test.


Cheers,

Brendan

[zesterer]

I don't know if this is still relevant, but for kernel threads I "store" the registers on the stack of the old thread, and then only swap out the stack point of the old thread for the stack point of the new thread. That way, when I pop off what the machine thinks is the old thread's registers from the stack, it ends up in the new thread.

Code: Select all

timer_interrupt_handler:
    PUSH_ALL_REGISTERS
    sp = scheduler_get_next_stack(sp);
    POP_ALL_REGISTERS
    iret

[LtG]

Even for kpanic; the goal is to continue for long enough to provide the user with relevant information (e.g. the fact that the panic was caused by an NMI and some details of what the CPU was doing when it happened) and stop other CPUs from running (and making a mess of things); and crashing (e.g. due to NMI-SMI-NMI) before you've done these things would be undesirable.

Yes, but if you are not going to return from the NMI then the solutions in the "NMI-SMI-NMI" thread should work without issues. Alternatively one could attempt to make the NMI handler "idempotent" so it doesn't matter how many times it's started it still works, assuming at least one of those NMI handlers gets to run to completion..

Some servers come with an "NMI button"; so that if an administrator notices an OS has locked up (e.g. maybe something like a spinlock that's spinning forever with IRQs disabled) they can press the button to get some idea of where the problem is (or get some information that makes a bug report better than useless). The other common cause is watchdog timers; which is basically an automated "tell me what the OS was doing if it locks up" alternative to the NMI button (e.g. normally the OS updates a timer regularly to stop it from expiring; and when the OS locks up it doesn't update the timer, so the timer expires and sends NMI).

Note: For hardware errors (e.g. memory corruption), I think most of it has been shifted to the machine check system in modern 80x86 computers. Unfortunately, if you go digging in modern chipset datasheets you'll still find (e.g.) registers that control things like "if FOO happens; send NMI or #SERR or SCI" and won't have any idea how the firmware actually configured the chipset (and can't easily create a list of things that cause NMI for a specific chipset).

For testing; one thing OS developers could really benefit from is an emulator designed for fault emulation and fault injection. Even basic things (e.g. checking if your "software RAID" layer actually does recover from hard drive failures properly) are excessively hard to test.

At least for the time being I'm not intending to have special support for an NMI button, but the generic NMI handler will eventually record a relevant error message. WD timer I don't yet have, but have thought about it, but in the case that it actually fires an interrupt I likely want to kpanic because it should never happen, so something is terribly wrong on a logical level if the WD timer fires.

I actually meant hardware generated NMI's that haven't been requested by the OS (eg. WD timer), and was thinking that memory corruption is likely handled by MCE these days, so wasn't sure if there's anything besides NMI button and WD timer that would actually trigger an NMI.

As for testing, I think one of the best ways would be to test with unit/integration tests and mocks to generate the NMI's or in general weird behavior. It probably gives you the best control and you can run all your tests as part of your compile/build process, I'm not a huge fan of manual testing, it's too unreliable in practice.

Though there are things that I don't know how to reasonably test, I don't really want to recreate a "virtual CPU" code for a mock object so I can test against it, but for a lot of stuff I think testing is quite reasonable. It adds dev time to create all the tests and mocks but it also decreases time spent debugging and hopefully produces better code.