How to set trigger mode and polarity in local APIC LVT?

[songziming]

I'm setting local APIC LVT for NMI.

I got information from MADT (Local APIC NMI Structure) that, NMI connects to all processors' LINT1, level sensitive, default polarity. So I should set LINT1's delivery mode as NMI, and trigger mode as level, right?

But Intel SDM says (11.5.1, inside item "Trigger Mode"):

When the delivery mode is NMI, SMI, or INIT, the trigger mode is always edge sensitive.

Software should always set the trigger mode in the LVT LINT1 register to 0 (edge sensitive). Level-sensitive interrupts are not supported for LINT1.

So that's conflict from MADT, which one should I listen to?

Also, what's the default trigger mode and default polarity of all kinds of interrupts?

Thanks.

[Octocontrabass]

I got information from MADT (Local APIC NMI Structure) that, NMI connects to all processors' LINT1, level sensitive, default polarity.

Are you sure? That doesn't sound right. Can I see a copy of your MADT?

Also, what's the default trigger mode and default polarity of all kinds of interrupts?

It depends on what bus the device is attached to. ISA is edge-triggered active high, PCI is level-triggered active low, and you can worry about other buses if you ever encounter them.

[songziming]

I found a mistake in my code, turns out NMI have default trigger mode and default polarity. My bad

MADT LOCAL APIC NMI content: processor-uid=255, local-apic-lint=1, flags=0x0.

So now my question is, how to determine the default trigger and polarity?

I also checked linux kernel source code. It doesn't check MADT, just set LINT0 as ExtINT, and LINT1 as level-trigger NMI. Is that config same for every PC?

[rdos]

NMI is a processor feature so no reason to check MADT.

[Octocontrabass]

MADT LOCAL APIC NMI content: processor-uid=255, local-apic-lint=1, flags=0x0.

That still doesn't look right. Can you share your MADT?

I also checked linux kernel source code. It doesn't check MADT, just set LINT0 as ExtINT, and LINT1 as level-trigger NMI. Is that config same for every PC?

I don't think it's guaranteed to be the same for every PC, but there's no reason why it would ever be different.

NMI is a processor feature so no reason to check MADT.

Modern processors don't have a dedicated NMI pin...

[rdos]

Modern processors don't have a dedicated NMI pin...

The NMI PIN never was used for anything significant in PCs, so that's hardly relevant. The interesting part about NMI is the ability to trigger it in the OS kernel to put all processor cores into known states. NMI ignores the interrupt flag, and so can used even when cores have crashed completely.

[Octocontrabass]

The NMI PIN never was used for anything significant in PCs, so that's hardly relevant.

This thread is about configuring the APIC LVT for the NMI pin, or whatever it is modern CPUs have instead of a physical NMI pin. NMI IPIs are hardly relevant.

[rdos]

The NMI PIN never was used for anything significant in PCs, so that's hardly relevant.

This thread is about configuring the APIC LVT for the NMI pin, or whatever it is modern CPUs have instead of a physical NMI pin. NMI IPIs are hardly relevant.

I'll just disagree. The MADT has no usable info on NMI since NMI is not a motherboard issue, but a CPU issue. The functionality of NMI is specified in the processor manuals, and so how it is implemented on motherboards is irrelevant. In the best case, MADT has the correct info, but apparently Linux doesn't trust this info and instead rely on processor documentation on NMI.

It's a bit similar to ACPI processor configuration and tuning processor speeds. Linux and probably Windows too doesn't use this info, but rather use processor drivers instead.

[nullplan]

So I've just read the ACPI spec on the topic of the NMI source, and am left wondering what the information is supposed to tell me. I concur with rdos here, the information is just useless. The thing I would like to know (and would expect the motherboard vendor to tell me) is what an NMI means when it occurs. Because I want to know what to do. Not how to set it up in the LAPIC, that's what the CPU manual already tells me. And in doubt I trust Intel over Dell any day of the week. I especially trust Intel to know more about Intel CPUs than Dell.

As for what to do, panic seems to be the most common answer. So trigger mode and polarity hardly matter; the interrupt will not re-occur, or rather, it will not matter.

[songziming]

MADT LOCAL APIC NMI content: processor-uid=255, local-apic-lint=1, flags=0x0.

That still doesn't look right. Can you share your MADT?

This is my MADT content:

Code: Select all

madt flags=1
madt loapic_addr=fee00000
Processor Local APIC, processor-id=0, apic-id=0, flags=0x1
Processor Local APIC, processor-id=1, apic-id=1, flags=0x1
Processor Local APIC, processor-id=2, apic-id=2, flags=0x1
Processor Local APIC, processor-id=3, apic-id=3, flags=0x1
IO APIC, io_apic_id=0, addr=fec00000, global_system_interrupt_base=0
Interrupt Source Override, bus=0, source=0, global_system_interrupt=2, flags=0x0
Interrupt Source Override, bus=0, source=5, global_system_interrupt=5, flags=0xd
Interrupt Source Override, bus=0, source=9, global_system_interrupt=9, flags=0xd
Interrupt Source Override, bus=0, source=10, global_system_interrupt=10, flags=0xd
Interrupt Source Override, bus=0, source=11, global_system_interrupt=11, flags=0xd
Local APIC NMI, processor-id=255, flags=0x0, lint=1

And if you want raw content (every two digits represent a byte):

Code: Select all

4150494390000000014b424f4348532042585043202020200100000042585043010000000000e0fe
010000000008000001000000000801010100000000080202010000000008030301000000010c0000
0000c0fe00000000020a0000020000000000020a0005050000000d00020a0009090000000d00020a
000a0a0000000d00020a000b0b0000000d000406ff000001

The NMI PIN never was used for anything significant in PCs, so that's hardly relevant.

I've heard that some servers have physical button to trigger NMI. Some machine also have watchdog that can send NMI if system doesn't respond. If system freezes, sending a NMI can restart the kernel, since NMI cannot be masked. On PC, if system doesn't respond, we can just restart the machine, so NMI is not that useful.

[songziming]

So I've just read the ACPI spec on the topic of the NMI source, and am left wondering what the information is supposed to tell me. I concur with rdos here, the information is just useless. The thing I would like to know (and would expect the motherboard vendor to tell me) is what an NMI means when it occurs. Because I want to know what to do. Not how to set it up in the LAPIC, that's what the CPU manual already tells me. And in doubt I trust Intel over Dell any day of the week. I especially trust Intel to know more about Intel CPUs than Dell.

As for what to do, panic seems to be the most common answer. So trigger mode and polarity hardly matter; the interrupt will not re-occur, or rather, it will not matter.

It seems Intel just provide NMI as a tool and let vendors make use of it. It's just a pin, Intel doesn't care what NMI means.

Vendors like Dell connects NMI to watchdog or physical button. NMI just means system is not reponding, and NMI handler needs to do a soft restart.

I don't know any other use of NMI, and my OS probably won't be used on such server. So panic is fine.

[rdos]

Yes, panic is fine, but my OS use NMI to trigger panic. However, panic in a debug environment means all cores are put into known states, and if you have a proper input device (PS/2 keyboard or serial connection to keyboard), you can inspect what CPU cores were doing when the panic happened, and single step the code in emulation mode.

I know no other reason to issue hardware NMI other than as a response to fatal errors. A watchdog timer should trigger RESET directly rather than NMI.

[nullplan]

Vendors like Dell connects NMI to watchdog or physical button.

Possibly. I would have now expected the ACPI information, being filled out by vendors like Dell, to contain information on what the NMI is. But it doesn't. The ACPI information tells me how to setup the LAPIC, and apparently does it wrong, so major OSes ignore the information. There is one other bit of info, the NMI Source structure. But it uselessly only tells me about GSI numbers that are shanghaied into NMI duty. This is useless because if the pin cannot be used, it will not be given as interrupt pin for any device, right?

Not to mention that interrupt pins matter less with each passing day, as x2APIC and MSI-X proliferate.

Yes, panic is fine,

OK, if Linux, Windows, RDOS, and toaruos all agree, then it can't be that wrong. I'll put a panic there.

[Octocontrabass]

And if you want raw content (every two digits represent a byte):

Huh! It does indeed say every local APIC's LINT1 is connected to NMI, with the default polarity and trigger mode. I wasn't expecting that.

Datasheets have trouble agreeing on exactly what the default should be, but the only reasonable default is active-high, edge-triggered - same as what Linux does.

I would have now expected the ACPI information, being filled out by vendors like Dell, to contain information on what the NMI is. But it doesn't.

It might be hiding in SMBIOS instead of ACPI. (Or it might be vendor-specific; Dell has a lot of stuff like that.)

OK, if Linux, Windows, RDOS, and toaruos all agree, then it can't be that wrong. I'll put a panic there.

Linux has the option of ignoring unexpected NMIs. I'm not sure how useful it is for modern hardware, but I've seen at least one ancient BIOS that "supports" non-parity RAM in its NMI handler...

[feryno]

Linux uses NMI as watchdog to detect kernel deadlocks. Instead of immediately restarting there is a message (kernel panic) printed to a console (and saved into a log) so you know that something ugly happened. It is possible to disable NMI watchdog using boot parameter nmi_watchdog=0.
Under Microsoft I saw NMI usage as IPI when hyper-v is running. So sending IPIs using APIC ICR to communicate among CPUs. I saw it since beginning of hyper-v (win server 2008). But the Microsoft usage is off topic concerning this thread because using APIC ICR, not APIC LINT / LVT. These NMI IPIs are sources of big problems to solve when developing hypervisor for Intel CPUs with nesting capabilities running as a parent and hyper-v as a child. The most challenging situations are when such NMI IPIs are delivered when parent hypervisor is executing its final vmresume instruction - postponing the NMI delivery to next vmexit/vmentry lead always into massive hangs and delivering them at current vmresume (where current RIP in parent hypervisor is now pointing) requires some "magic" steps and modifying RIP to point few instructions backward. Under Intel CPUs there is not convenient way to mask off NMIs so the NMIs are delivered from child hypervisor while parent hypervisor is running (from one CPU/core where child = hyper-v is running using APIC IPI into another CPU/core where parent hypervisor is running). Under AMD there is easy way to mask off NMIs when virtualization enabled (using the CLGI instruction, using VMCB.VGIF bit, and also every vmexit masks off NMIs because clearing GIF global interrupt flag). Under AMD I encountered also problems (no matter easier way to virtualize GIF and disabled NMI delivery when parent hypervisor running) as my parent hypervisor uses NMI IPIs to communicate among all CPUs/cores and Linux kernel uses APIC LINT and LVT as a source of NMIs which seems to be fired too much quickly which is OK on hardware but cannot be handled so fast under virtualization - I need to intercept NMIs and the vmexit/vmrun lasts more CPU cycles due to virtualization than 2 subsequent Linux NMIs (APIC LINT and LVT) supposed to execute quickly enough on hardware so I have to use the above boot param nmi_watchdog=0. Maybe I have another problem but the above boot param helps me.
Yes that's true, some servers have a button to send NMI. Perhaps such NMI could be useful if the server hangs. A server could have not only a NMI button but also a NMI LED diode (maybe if a crash is too severe that even no possibility to print to a console, then still the LED diode gives a clue what happened).