Page 1 of 1
IPI from withing network interrupt
Posted: Thu Apr 09, 2015 2:51 am
by kemosparc
Hi,
I have a problem related to issuing IPIs upon reception of a packet.
I have a network driver for e1000 (int 11), and I want to issue an IPI to a core to handle it, so I send an IPI to the AP when a packet is received at the end of handling int 11.
In Qemu the IPI does not go through, meaning that bit 12 of ICR0 get cleared after I send the IPI which means that the IPI was accepted by the recipient core but the handler does not fire. When I comment the IPI sending code from within the interrupt handler of the network card and move the same code inside the PIT handler the IPI handler on the AP fires normally with no problems.
Surprising, the setup works perfectly on bochs. So, when I moved to bochs for more debugging info I could not replicate the problem, it works as designed exactly.
In both cases the network interrupt fires and I can print ticks of the number of packets received.
The only difference is that I use on bochs a bridged network (on my physical interface), and I use on Qemu a virtual interface virbr0, but I don't think that this could be a problem, could be?
Could this be related to interrupt priorities ?
Kindly let me know if any one could think of something wrong I might be doing related to qemu.
Thanks a lot,
Karim.
Re: IPI from withing network interrupt
Posted: Thu Apr 09, 2015 4:23 am
by Brendan
Hi,
The only plausible explanation that I can think of (excluding bugs in either your OS or the emulator); is if the original IRQ used the IO APIC and the IPI is using "send to lowest priority". In this case P6 CPUs (which if I remember correctly is what Qemu mostly emulates despite what it says the CPU is) have a "focus processor" feature where the CPU that's already handling that interrupt vector will accept the interrupt before other CPUs (with lower priority) do. Of course I don't know if you're using the IO APIC or if you're sending the IPI with "send to lowest priority" (and I suspect you're using PIC and fixed delivery, so I assume this is unlikely to be the problem).
Cheers,
Brendan
Re: IPI from withing network interrupt
Posted: Thu Apr 09, 2015 4:30 am
by kemosparc
Thanks,
The weird thing is that bochs works, and I always want to have the problem replicated on bochs as it is easy to get debug info.
So how can I make qemu emulate other processor types, may be if I play with the processor type and the behaviour start to change I might get more understanding of what is going on.
Thanks,
Karim.
Re: IPI from withing network interrupt
Posted: Thu Apr 09, 2015 6:10 am
by kemosparc
Hi,
Okay I did some digging in qemu and came up with some info, I hope that someone can help me with the info that I cannot explain
Anyways, I kept on playing around with Qemu's code and here what happens.
The problem is related to this code inside qemu hw/intc/apic.c:
Code: Select all
#define clzh(v) ((v) ? clz32((uint32_t)(v) << 16) : 16
static int apic_irq_pending(APICCommonState *s)
{
int irrv, ppr;
irrv = get_highest_priority_int(s->irr);
if (irrv < 0) {
return 0;
}
ppr = apic_get_ppr(s);
if (ppr && (irrv & 0xf0) <= (ppr & 0xf0)) {
return -1;
}
return irrv;
}
/* return -1 if no bit is set */
static int get_highest_priority_int(uint32_t *tab)
{
int i;
for (i = 7; i >= 0; i--) {
if (tab[i] != 0) {
return i * 32 + apic_fls_bit(tab[i]);
}
}
return -1;
}
* Find first bit starting from msb */
static int apic_fls_bit(uint32_t value)
{
return 31 - clz32(value);
}
What happens is that in the working case when the IPI is initiated from inside the PIT interrupt hander (int 32 - IRQ0) the get_highest_priority_int returns 64 which is the interrupt number of AP, and in the other case it returns -1. I printed the s->irr (an array of 8 uint32_t)and it was "0 0 1 0 0 0 0 0" in the working case and all zeros in the faulty case.
The get_highest_priority_int returns -1 when no bit is set. My question is that what am I not doing for this bit to be set while I am in the e1000 handler (int 43 - IRQ11)?
Thanks,
Karim.
Re: IPI from withing network interrupt
Posted: Thu Apr 09, 2015 7:49 am
by kemosparc
Hi,
For some reason Qemu cannot write bit 3 (int 64) into the IRR of the AP while it is serving the network.
What could be the cause of that ?
Thanks,
Karim.
Re: IPI from withing network interrupt
Posted: Thu Apr 09, 2015 11:29 pm
by Brendan
Hi,
kemosparc wrote:For some reason Qemu cannot write bit 3 (int 64) into the IRR of the AP while it is serving the network.
What could be the cause of that ?
Which IRR? Each CPU has a local APIC, so if there's 4 CPUs there's 4 different IRRs. It's possible that you're looking at the wrong IRR, and also possible that the interrupt was delivered to a CPU (and the corresponding bit in that CPU's IRR was cleared when bit in its ISR was set).
Note: I still don't know what you're doing. E.g. if you're using "fixed delivery" for the IPI (and if you are, where you get the target APIC ID from), or if/how you've setup the local APIC's in all the CPUs (e.g. whether you've set the local APIC's "task priority register" properly).
kemosparc wrote:So how can I make qemu emulate other processor types, may be if I play with the processor type and the behaviour start to change I might get more understanding of what is going on.
This might help (or might not help and just be a useless distraction that has nothing to do with the issue).
You've seen the local APIC code - did you see anything that is effected by the emulated CPU type? I expect it's modelled on a "generic P6" local APIC and nothing changes that; in the same way the chipset is based on an old Pentium chipset and isn't effected by CPU type either.
Cheers,
Brendan
Re: IPI from withing network interrupt
Posted: Fri Apr 10, 2015 5:49 am
by kemosparc
Hi,
Thanks a lot for the reply and for the link, I will try different CPU models as soon as I have access to my desk
Answering your questions, I use a VM with 2 processors 1 BSP and 1 AP. I use ID 1 to send fixed IPI to the AP from the BSP. I also tried using the shorthand "all excluding self" by setting bits 18 and 19 in icr0 and still have the same problem.
I don't think that it is a problem with the core ID as when I send from inside the PIT handler the same code works, with the same core ID I use, I did not change anything.
What I think is that it is a problem with the setting of the NIC e1000 interrupt gate. What I did is that I debugged qemu while running my OS (I did not debug my OS within qemu through remote GDB, no I started Qemu using GDB and added some print statements) and the problem is that the second if condition "else if (apic_irq_pending(s) > 0) " never fires when I call the IPISend from within the E1000, but when I do from the PIT it fires on every IPI sending !!!! In the working setup apic_irq_pending(s) returns 64 which is the interrupt number of the IPI.
Code: Select all
/* signal the CPU if an irq is pending */
static void apic_update_irq(APICCommonState *s)
{
CPUState *cpu;
if (!(s->spurious_vec & APIC_SV_ENABLE)) {
return;
}
cpu = CPU(s->cpu);
if (!qemu_cpu_is_self(cpu)) {
cpu_interrupt(cpu, CPU_INTERRUPT_POLL);
} else if (apic_irq_pending(s) > 0) {
// This if never fires when I call the IPI from the E1000 handler
cpu_interrupt(cpu, CPU_INTERRUPT_HARD);
}
}
Thanks,
Karim.