x2APIC: INIT IPI delivered but #GP raised by sender (bochs)

[Qeroq]

Hello,
I am currently working on extending Hydrogen with x2APIC support and while the modifications are quite straightforward I ran into a problem when trying to start up the APs using bochs (Any other emulator/vm that supports the x2APIC?):

When I send an INIT IPI to an AP, the IPI is delivered (as the bochsout.txt tells me), but a #GP is raised on the sender with the ESR clear. When I send the INIT IPI to all CPUs excluding self using the destination shorthand and then send an SIPI to the first AP, it properly runs its startup code as it should, but still the #GP is raised on the ICR MSR write. When I try to send an IPI to the current processor itself (fixed delivery mode), the interrupt is delivered correctly and no exception is caused.

The ICR value I use for the INIT IPI is 0x00000001_00004500, i.e. INIT delivery mode, physical destination mode, no shorthand, level assert, edge trigger and the APIC ID of the first AP (1).

Any idea what could be wrong? Any pitfalls or quirks of the x2APIC or of bochs implementation of these that I'm missing? When code is required, feel free to ask for the relevant parts; I will upload the modified version to GitHub as soon as I cleaned up the mess left behind by debugging and refactoring.

Regards, Farok

[Nable]

may be you should turn on debugging for cpu? Bochs is (if you allow it) rather verbose about problems.

http://forum.osdev.org/viewtopic.php?p=206304#p206304
> Bochs has debug prints for every faults it can have. Just enable "debug: ignore, cpu0=report" and you will see all possible prints including reasons for all faults.

[Qeroq]

"debug: ignore, cpu0=report" wasn't accepted by my version of bochs (2.5.4, compiled from SVN), so I went for "debug: action=report" and searched for the place the exception was raised (grep -v XGUI bochsout.txt | grep -B 10 -A 10 "exception(0x0d)"):

Code: Select all

00065446930d[CPU0 ] page walk for address 0x0000000000001000
00065447995i[CPU0 ] [65447995] Stopped on MAGIC BREAKPOINT
00065447996d[CTRL ] searching for component 'keyboard_mouse' in list 'bochs'
00065447996d[CTRL ] searching for component 'mouse' in list 'keyboard_mouse'
00065447996d[CTRL ] searching for component 'enabled' in list 'mouse'
00065447996d[KBD  ] PS/2 mouse disabled
00065447996d[CTRL ] asking for next debug command
00065447996d[CTRL ] received next debug command: 'c'
00065448031d[CPU0 ] WRMSR: write 00000001:00004500 to MSR 830
00065448031i[APIC1] Deliver INIT IPI
00065448031d[CPU0 ] exception(0x0d): error_code=0000
00065448031d[CPU0 ] interrupt(): vector = 0d, TYPE = 3, EXT = 1
00065448031d[CPU0 ] interrupt(long mode): INTERRUPT TO SAME PRIVILEGE
00065448031i[CPU1 ] cpu software reset
00065448031i[APIC1] allocate APIC id=1 (MMIO enabled) to 0x00000000fee00000
00065448031i[CPU1 ] CPU[1] is an application processor. Halting until IPI.
00065448031i[CPU1 ] CPUID[0x00000000]: 0000000b 68747541 444d4163 69746e65
00065448031d[CTRL ] searching for component 'cpu' in list 'bochs'
00065448031d[CTRL ] searching for component 'n_threads' in list 'cpu'
00065448031i[CPU1 ] CPUID[0x00000001]: 00000633 01010800 00202028 07cbfbff
00065448031i[CPU1 ] CPUID[0x00000002]: 00000000 00000000 00000000 00000000

Note the line with exception(0x0d): error_code=0000 after the delivery of the INIT IPI. Also note the write of 00000001:00004500 to MSR 830 and CPU1 being initialized afterwards. I think its a bug in the x2APIC simulation.

Edit: Really seems to be a bug in bochs. I checked the code and found the following:

Code: Select all

bx_bool bx_local_apic_c::write_x2apic(unsigned index, Bit64u val_64)
{
  Bit32u val32_lo = GET32L(val_64);

  index = (index - 0x800) << 4;

  switch(index) {
// SNIP

  // handle full 64-bit write
  case BX_LAPIC_ICR_LO:
    send_ipi(GET32H(val_64), val32_lo);
    break;

// SNIP

  default:
    BX_DEBUG(("write_x2apic: not supported apic register 0x%08x", index));
    return 0;
  }

  if (GET32H(val_64) != 0) // upper 32-bit are reserved for all x2apic MSRs 
    return 0; 
  write_aligned(index, val32_lo);
  return 1;
}

Looks like the IPI is sent, but then the upper 32 bits of the value are checked. If they are not clear, zero is returned, just as if the register never existed (which would cause an #GP). This explains why the local IPI worked without a #GP, because the destination, i.e. the upper 32 bits, were clear (APIC ID of zero). Additionally it seems like write_aligned handles the ICR write (lower one of the DWORDs) even again, but that time with a destination of (higher dword >> 24) & 0xFF. I'll check out the newest commit of bochs, and see if its still there; if it is, I'll file a bug report.

Edit 2: Yep, I replaced the "break;" with a "return 1;" after "send_ipi(GET32H(val_64), val32_lo);" and it worked.

[stlw]

Edit 2: Yep, I replaced the "break;" with a "return 1;" after "send_ipi(GET32H(val_64), val32_lo);" and it worked.

I will merge the fix into Bochs SVN now

Stanislav