[SOLVED] Interrupts with IOAPIC - BSP ok, no AP

[sentientnz]

Looking for the trick to solve my annoying little problem

I have 'started up' my application processors into a sort of 'holding pattern' and initialized the IOAPIC.

When the IOAPIC is mapped to the BSP, I then enable a single irq (IRQ1 - keyboard) and enter an idle loop. If i press a key, I get a message from the interrupt handler.

If I map the IOAPIC to any AP, and perform the same procedure, no interrupts fire. I have tried the same with IRQ0 (timer) - still no interrupts. The AP's don't appear frozen - they are still looping through their 'idle' (which draws a character on the screen (A through Z) so I know they have frozen in an interrupt handler etc..).

Is there anything that needs to be enabled on the AP that might be enabled on the BSP when using GRUB loader?

I have checked that interrupts are enabled (STI), the APs have the same IDT loaded.

Ideas appreciated.

[Brendan]

Hi,

Ideas appreciated.

A quick "from the top of my head" checklist:

• PIC masked correctly
• Spurious interrupt vector and the "enable" flag (both in the same register) in each local APIC
• "Task priority register" setup correctly in each local APIC
• "Destination Format Register" and "Logical Destination Register" setup correctly in each local APIC (if using logical delivery of the IRQ and not physical delivery)
• IOAPIC setup correctly to deliver the IRQ/s to the intended CPU (and not just the BSP if you intended to receive IRQs with AP CPUs)
• STI in the APs

Cheers,

Brendan

[gerryg400]

My sequence looks very much like Brendan's. Some added thoughts..

Can you get APIC timer ints on the APs ? Start there.

Or try an EOI to kick things off.

One problem I had early on was that I didn't have locking around my IOAPIC accesses and that got it into some some kind of bad state.

[sentientnz]

In response:

1) PIC seems to be masked correctly. I have remapped PIC to 0x20->0x2F, and the IOAPIC to 0x30->0x4F so i can be sure which one fired. After the PIC is disabled and IOAPIC enabled, I only get IOAPIC interrupts.
2) Spurious Vector and enable flag are set on each AP.
3) Task Priority is set on each AP.
4) DFR and LFR are set (although I am using physical destination anyway).
5) ...
6) STI is called after setup on each AP

5) Setup the IOAPIC for AP delivery is a bit tricker - it 'seems' to be setup although I have not recieved a single IOAPIC-generated IRQ on an AP. It delivers fine to the BSP - PIT Timer & keyboard IRQs are delivered without problem. I believe that if I want the interrupts delivered to APICID=1 (an AP) rather than APICID=0 (my BSP) that I just need to change the destination in the IOREDTBL - which I have done... No interrupts are delivered

I tried what gerryg400 mentioned (getting Local APIC timer working) - Have run into a new snag. I programmed the LAPIC timer in periodic mode and recieved 1 interrupt from each AP, then nothing The processors have returned from the interrupt handlers - the 'heartbeat' is still working - but no further interrupts (The BSP is the same).

I tried moving the IOAPIC interrupts back to the BSP with the Local APIC timers enabled - same problem. I recieved one keyboard interrupt from the IOAPIC, then one LAPIC timer interrupt and then no more - it also blocks delivery of further IOAPIC interrupts.

It appears I am not handling the EOI for the LAPIC correctly. At present, I just do:

volatile u32 *apic_eoi = (u32 *)0xFEE000B0;
*apic_eoi = 0;

Am I missing something?

PS: I don't have paging enabled

[sentientnz]

I have fixed the issue with the Local APIC timer not firing - seems that GCC was doing something with the EOI memory write.
I added the EOI write to the assembly code and it works fine now. I can get LAPIC Timer Interrupts on all 4 cpu's (BSP + 3x AP).

Still not having much luck with redirecting the IOAPIC though. IOAPIC irqs are working fine on the BSP with LAPIC Timer interrupts.

For reference:
On the BSP I use the following entry for IOREDTBL
flags = 0x0000 0000 0001 0020;

To redirect IRQs to APICID=1, I change it to:
flags = 0x0100 0000 0001 0020;

[gerryg400]

You need to be very careful with GCC. It is allowed to move instructions around and it will perform unexpected optimisations. For example in my code I had.

Code: Select all

    __cli();
    apic_write(LAPIC_EOI, 0);

GCC would swap those 2 instructions.

What exactly was GCC doing ? Don't ignore the problem because it seems to have gone away, make sure you understand the problem and understand precisely how your solution works. In a multicore kernel you need to be especially careful about these things.

[gerryg400]

Code: Select all

flags = 0x0100 0000 0001 0020;

Bit 16 is the masked bit. So that means that int 0x20 is masked. Is that what you want ?

[Nable]

Note: you can use

Code: Select all

asm volatile ("");

to prevent gcc from reordering your operators. Just insert it between two lines than must be kept in a strict order (lots of examples are in linux kernel code).

[sentientnz]

gerryg400 - yes, i initialize them disabled and later on I enable them with:

static void ioapic_write(void *base, u32 offset, u32 val)
{
((volatile u32 *)base)[0] = offset;
((volatile u32 *)base)[4] = val;
}

static u32 ioapic_read(void *base, u32 offset)
{
((volatile u32 *)base)[0] = offset;
return ((volatile u32 *)base)[4];
}

static int ioapic_enable_irq(u8 irq)
{
u32 flags0 = ioapic_read(apic_base, (irq * 2) + 0x10);
ioapic_write(apic_base, (irq * 2) + 0x10, flags0 & ~0x00010000);
return 1;
}


Just to check, I unmasked all the IRQs at setup time, and tried it again. When mapped to the BSP, IRQ's seem to flow out (IRQ0 in particular) - still nothing on any AP.

[gerryg400]

I can only think of 3 possibilities.

1. AP doesn't have STI
2. code that writes to upper 32 bits of IOAPIC redirection table is broken.
3. The CPU's have a funny numbering scheme. e.g. 0, 4, 8, 12 etc.

I'm thinking it's 2. Try reading back the entire 64 bit value that you write to the IOAPIC redir reg and make sure it's written correctly. Are ints still coming to the BP when you try to send them to the AP.

[sentientnz]

1) AP definately seems to have STI - LAPIC interrupts are occurring.

3) CPU numbering seems to be linear on QEMU (1,2,3,4) - I printf the APICID's on bootup. On my Dual-core Intel Core2, the cores are numbers 0,1,4,5 (presumably the hyperthreading 'cores' are 4 and 5).

2) Possibility - I have in 32-bit mode, so perhaps I am screwing up the upper dword writes. I'll double check the code.

Update:

I did a bit of testing regarding programming the upper dword of IOREDTBL entries. It definately seems to be changing the destination. If the highest byte of the upper dword is changed from 0x00 I no longer recieve interrupts on the BSP (or the AP's).

[sentientnz]

I still don't know what is causing the problem with these interrupts not firing on AP's but I have found a workaround.

I was mapping all the IOAPIC interrupts to vectors 0x20->0x2F - worked fine on BSP, but nothing on AP

To try an isolate the problem, and ensure I was getting interrupts from the PIC 8259, I moved the IOAPIC to vectors 0x30->0x3F.
All of a sudden, I can get interrupts on all processors (BSP and 3x AP). I thought I had found the cause of the problem (PIC / IOAPIC conflicting).

I then moved the PIC vectors to 0x30-0x3F as well (leaving the IOAPIC in the same range (0x30-0x3F) and I still get interrupts from the IOAPIC.

Leaving the PIC at the 0x30 range, and moving the IOAPIC back to 0x20-0x2F - No interrupts on AP's again.

?!?!?!?!??!!?



UPDATE:

Problem is solved.
Because the interrupts were mapped to 0x20->0x2F, and the TASKPRI was 0x20 - well, duh
Changed TASKPRI = 0x00, remapped IOAPIC interrupts to 0x20-0x2F and all works as expected now.