x2Apic Lint Config

[johnsa]

Hi,

So i've gotten my SMP setup working, all the cores are trampolining into long mode and configuring their PAT, local apic, mtrrs etc and the locking is in place.
I've followed Brendan's notes about the INIT-SIPI-SIPI procedure with the relevant delays, synchronization etc.

My local apic code supports both xapic and x2apic mode. All my emulators only run xapic mode while my real machines run x2apic. If I leave the init in xapic all runs as planned, the second I switch to x2apic mode (and on real h/w) configuring the lint0 and lint1 in any way seem to cause a GPF. (If I leave them masked and not setup in any way everything runs through as planned - with one small catch.. the timer (pit) no longer runs at all so the delays don't work.. I can comment those out and purely by good fortune the cores even on real h/w still fire up).

I am assuming the following:
1) From the MADT entries I find the local apic and x2apic entries which I use during configuration.
2) I find the local apic NMI and x2apic nmi entries which tell me which of lint0 or lint1 should be configured as the nmi type (with settings for level/edge, active low/high taken from the respective entry).
3) I then assume that the remaining lint should be configured as extINT (to handle the 8259a while I still haven't configured the ioapic).

So assuming the MADT entry tells me LINT1 is the NMI:

Code: Select all

	mov rcx,MSR_IA32_EXT_XAPIC_LVT_LINT0
	rdmsr
	mov eax,0x2700   ;vector=0, extINT,edge, active low
	wrmsr
	mov rcx,MSR_IA32_EXT_XAPIC_LVT_LINT1
	rdmsr
	mov eax,0x2400  ;vector=0, NMI, edge, active low
	wrmsr

Using the above exactly values avoids the GPF, but the PIT/timer still doesn't fire anymore..
Note I also only configure this block for the BSP (as only one processor should respond to the legacy 8259a stuff and that would be the BSP).

4) I also can't quite figure from the manuals exactly which interrupts or IRQs should then also send the local apic EOI (Is it all IRQ handlers) or IRQ handlers, softints and LVTs excluding (Performance,Thermal etc)
The manual says it shouldn't be sent for types SMI,NMI,extInt,startup etc but since those registers above ignore any vector I assume they don't directly have a handler either?

If I for example put the x2apic eoi on the timer IRQ (as below) then once again i start getting a GPF.

Code: Select all

	xor eax,eax
	xor edx,edx
	mov rcx,MSR_IA32_EXT_XAPIC_EOI
	wrmsr

5) I could just leave these setup a different way and forget about using the PIT timer at this point? Perhaps it's too defunct emulation. I could use local apic timer during SMP startup, then afterwards finalize the ioapic/hpet setup at which point legacy RTC irqs and PIT are history anyway.

6) Assuming the ioapic is configured and 8259a is history, what settings should the lint(0/1) be configured for then? (I assume still the same and the ioapic goes directly to the lapic.. bypassing the lint pin?)

[Brendan]

Hi,

My local apic code supports both xapic and x2apic mode. All my emulators only run xapic mode while my real machines run x2apic. If I leave the init in xapic all runs as planned, the second I switch to x2apic mode (and on real h/w) configuring the lint0 and lint1 in any way seem to cause a GPF. (If I leave them masked and not setup in any way everything runs through as planned - with one small catch.. the timer (pit) no longer runs at all so the delays don't work.. I can comment those out and purely by good fortune the cores even on real h/w still fire up).

I am assuming the following:
1) From the MADT entries I find the local apic and x2apic entries which I use during configuration.
2) I find the local apic NMI and x2apic nmi entries which tell me which of lint0 or lint1 should be configured as the nmi type (with settings for level/edge, active low/high taken from the respective entry).
3) I then assume that the remaining lint should be configured as extINT (to handle the 8259a while I still haven't configured the ioapic).

If ACPI doesn't mention LINT0, don't touch it. There's at least 3 different ways that the master PIC's "INTR" line could be connected:

• Master PIC's "INTR" line connected to a local APIC input (e.g. LINT0)
• Master PIC's "INTR" line connected to an IO APIC input
• Master PIC's "INTR" line not connected when the IO APIC is being used (e.g. gated by an "IMCR"), where the legacy devices are connected directly to IO APIC inputs.

Using the above exactly values avoids the GPF, but the PIT/timer still doesn't fire anymore..

For ACPI; for legacy IRQs (not including PCI) you initially assume that "ISA IRQ 0" is connected to IO APIC input #0, "ISA IRQ 1" is connected to IO APIC input #1, etc (all the way up to "ISA IRQ 15" is connected to IO APIC input #15). Then you parse the "Interrupt Source Override" structures in ACPI's MADT/APIC table to determine cases where that initial assumption is wrong. Typically there is an "Interrupt Source Override" telling you that ISA IRQ 0 (the PIT) is actually connected to IO APIC input #2. For some computers (rare) the PIT isn't connected to the IO APIC at all and can't be used.

4) I also can't quite figure from the manuals exactly which interrupts or IRQs should then also send the local apic EOI (Is it all IRQ handlers) or IRQ handlers, softints and LVTs excluding (Performance,Thermal etc)
The manual says it shouldn't be sent for types SMI,NMI,extInt,startup etc but since those registers above ignore any vector I assume they don't directly have a handler either?

You'd send the EOI to the local APIC for all IRQs that Intel didn't list - e.g. for all interrupts from devices connected to the IO APIC, for the local APIC's internal sources (thermal sensor, performance monitoring, local APIC timer, etc), and for Inter-Processor Interrupts (including "send to self" IPIs).

If I for example put the x2apic eoi on the timer IRQ (as below) then once again i start getting a GPF.

Code: Select all

	xor eax,eax
	xor edx,edx
	mov rcx,MSR_IA32_EXT_XAPIC_EOI
	wrmsr

Are you sure you've initialised the local APIC properly (including enabling it and setting a spurious interrupt vector in the "spurious interrupt" register; and sure you're using "X2APIC" mode (and that it's not emulating XAPIC)? Also, are you sure the PIT's IRQ handler itself or its IDT entry aren't causing the GPF (note: because PIT is relatively simple you can test that its IRQ handler works by removing the EOI and using a software interrupt, or by using an IPI and leaving the EOI "as is").

I'd also recommend testing the local APIC with IPIs to make sure it's configured before worrying about IO APICs. Otherwise there's a higher risk of "2 problems masking each others symptoms" (that makes it much harder to figure out what's wrong).

5) I could just leave these setup a different way and forget about using the PIT timer at this point? Perhaps it's too defunct emulation. I could use local apic timer during SMP startup, then afterwards finalize the ioapic/hpet setup at which point legacy RTC irqs and PIT are history anyway.

This depends on whatever "minimum requirements" you decide to support. Don't forget that the local APIC timer can break (stop counting) when the CPU is using low power modes; so you need a fall-back for that. Nothing really guarantees that HPET is supported.

6) Assuming the ioapic is configured and 8259a is history, what settings should the lint(0/1) be configured for then? (I assume still the same and the ioapic goes directly to the lapic.. bypassing the lint pin?)

They'd be configured how ACPI tells you to configure them (including "left as found" if ACPI tells you nothing). Don't forget that if you're using the IO APIC and not using the PIC, the "extINT" is mostly irrelevant (and it's a very idea to attempt to use both PIC and IO APIC at the same time).


Cheers,

Brendan

[johnsa]

Once again, brilliant info!

I found my immediate error, the SMP trampoline was working 100%, once the AP core started up in kernel space it began to configure its own local apic, if the x2Apic mode was flagged by the BSP it merrily tried to access the MSRs .. without first doing a global enable and x2apic enable in the apic base msr... hence gpf.

For now I'm leaving the LINT settings as I find them (except for ensuring the NMI ones are configured as per ACPI per core).
On the BSP I leave lint0/1 unmasked (in case the pit is wired to the LINT pin so it can continue to run until i setup the ioapic),
and for the AP cores I leave them masked along with CMCI,Thermal,Perf,Timer etc as any 8259a emulation should only route to the BSP.