APIC timer - random triple fault problem

[r0bypasser]

Hello,
I've written some code which enables local APIC and APIC timer per each processor.
It works well in bochs, but testing in VMware, randomly causes triple fault from AP.

I've configured local APIC and APIC timer as follows:

<local APIC>
IA32_APIC_BASE_MSR[11] = 1 (APIC global enable)
SVR[7:0] = 70h (spurious vector 70h)
SVR[8] = 1 (software enabled)

<APIC timer>
Divide Configuration Register[3:0] = 3 (0011, set divide fector to 16)
TMR[18:17] = 1 (periodic mode)
TMR[16] = 0 (not masked)
TMR[7:0] = 30h (APIC timer vector at 30h)

I checked which interrupt was called when fault occurs, but i found nothing.
Is there any configurations that i mistaken?

[Nable]

You don't use counter register of apic timer, do you?
Also, you may have smth wrong in your IDT.

[r0bypasser]

I've been using processor's bus clock speed as ICR value.
I checked IDT again and found no problem but triple fault still occurs.

[bluemoon]

did any other fault occurred?

I suggest handle all 00-19 exception with kprintf to serial output, so you can see why triple fault.
ps. even nothing shown would mean an problematic IDT.

[r0bypasser]

i tested repeatedly, and found the reason why triple fault occurred!

i wrote the code which prints entries of IDT (index 00~1fh) in each processor.
after tested, i got the result some of AP's IDT (index 00~1fh) points null address.
and faulty processor of APIC id is randomly changed.

test result

but, it is strange... my AP initalization code is:

Code: Select all

	// EFLAGS.IF = 0 at this point

	// fill current processor's IDT entries
	// index 00~1fh, 30h (LAPIC timer), 70h (spurious interrupt)
	HalpSetDefaultInterruptVectors();

	// print IDT entries (index 00~1fh) in current processor
	KdpDumpProcessorIdt();

	// enable local APIC & APIC timer
	HalpInitializeLocalApic();
	HalApicInitializeTimer();

	// enable interrupt (EFLAGS.IF = 1)
	__asm sti

as shown, i print IDT entries in screen after IDT entries have filled.
so, value of IDT entries which i set cannot be zero.

but, the result is not...
i also tested in bochs and QEMU, but the problem only occurs in VMware.

why...?????

[xenos]

Could you post your AP startup code? I guess you send some Startup / Init / Init IPI sequence to wake up the APs? It's possible that some APs are not started up properly and don't run their initialization code at all.

I once had a very similar problem which showed up only on AMD SimNow!. It turned out that the APs started earlier than expected (before the second Init IPI) and jumped to the wrong address when they received the second Init IPI, causing a triple fault. I finally fixed it by checking whether the APs already started up before sending the second Init IPI.

[r0bypasser]

i used INITIPI-SIPI-SIPI to wake up APs.

Code: Select all

        // Send an INIT IPI (all-except-self)
        // Delay 10 Msc
        HAL_TRACE("Send INIT IPI\n");
        __asm mov dword ptr ds:[LAPIC_ICR0], 0c4500h
_WaitIpiPending0:
        __asm bt dword ptr ds:[LAPIC_ICR0], 0ch
        __asm jc _WaitIpiPending0
        HalPitWaitMillisecond(10);

        // Send Startup IPI (all-except-self)
        // Delay 200 usc
        // MP Init Real-Mode Code at (0x01 << 12)
        HAL_TRACE("Send Startup IPI\n");
        __asm mov dword ptr ds:[LAPIC_ICR0], 0c4601h
_WaitIpiPending1:
        __asm bt dword ptr ds:[LAPIC_ICR0], 0ch
        __asm jc _WaitIpiPending1
        HalPitWaitMillisecond(1);

        // Send Startup IPI (all-except-self)
        // Delay 200 usc
        // MP Init Real-Mode Code at (0x01 << 12)
        HAL_TRACE("Send Startup IPI (Wake up AP)\n");
        __asm mov dword ptr ds:[LAPIC_ICR0], 0c4601h
_WaitIpiPending2:
        __asm bt dword ptr ds:[LAPIC_ICR0], 0ch
        __asm jc _WaitIpiPending2
        HalPitWaitMillisecond(1);

in my code, each AP starts up at physical address 0x10000 and
initializes itself in function KiInitializeAP.

Code: Select all

VOID
KiInitializeAP(
        VOID *KernelStack)
{
        UCHAR ProcessorNumber;

        ProcessorNumber=HalGetProcessorId();
        if(!KernelStack)
        {
                // kernel stack allocation failed, halt system
                ...
        }

        KTRACE("Processor %d - Kernel Stack 0x%p\n", ProcessorNumber, KernelStack);

        KiProcessorBlock[ProcessorNumber].Self=&KiProcessorBlock[ProcessorNumber];
        KiProcessorBlock[ProcessorNumber].KernelStack=(ULONG)KernelStack;
        KiProcessorBlock[ProcessorNumber].StackSize=KERNEL_R0_STACK_SIZE;
        KiProcessorBlock[ProcessorNumber].ProcessorNumber=ProcessorNumber;

        // create kernel segments and locate real IDT/GDT
        KiSetupSystemTable();

        // initialize IDT entries and LAPIC timer
        HalpInitSystem();

        for(;;); // infinite loop
}

VOID
HalpInitSystem(
        VOID)
{
        // EFLAGS.IF = 0 at this point

        // fill current processor's IDT entries
        // index 00~1fh, 30h (LAPIC timer), 70h (spurious interrupt)
        HalpSetDefaultInterruptVectors();

        // print IDT entries (index 00~1fh) in current processor
        KdpDumpProcessorIdt();

        // enable local APIC & APIC timer
        HalpInitializeLocalApic();
        HalApicInitializeTimer();

        // enable interrupt (EFLAGS.IF = 1)
        __asm sti
}