OSDev.org

Posted: **Wed May 02, 2012 11:55 am**

I've a system with a peripheral PCIe card and I am trying to make it trigger MSI-X interrupt.
I've verified that my MSI setup works fine with the help of HPET.

I leave all my PCI bridges and entire fabric in the state it was left by the BIOS; i.e. I do not perform
bus enumeration and I don't touch bridge configuration spaces. The only thing I do is enable
MSI-X on the peripheral device and disable INTx emulation in the configuration space. I populate MSI-X table slot 0,
leaving all other vectors masked. My destination and address register values are correct (verified with HPET MSI)
so I am a bit stuck trying to figure out why the device is not sending out interrupts when it should.

I'd appreciate any ideas on what to investigate when trying to figure out why the MSI-X is not delivered. I'm a bit stuck on this one and have been following multiple investigation paths in parallel but it's nice to have some fresh ideas/suggestions.

Thanks

Posted: **Thu May 03, 2012 1:09 am**

I think you need to post code on how you setup MSI-X, and why you expect an interrupt from the network card. Most network cards will need extensive setup before they will generate interrupts.

Posted: **Thu May 03, 2012 3:35 pm**

I've done the card set-up (done all initial programming, card reset, PCIe configuration, MSI-X enabling, etc.) and it brings up the ports, LEDs turn on, link training completes, etc.

The card has a diagnostic command that I can use to generate events that should trigger MSI-X interrupts.

The way I set up my MSI-X table is as follows ( msix + 0 = low 32-bits of message address register, msix + 4 = high 32-bits of message address register, msix + 8 = message data register, msix + 12 = msi-x vector control (unmasked) )

entry 0: FEE00000.00000000.00000079.00000000

entries 1-0x80 are all set to 0 with only MASK bit set.

PCI configuration space has INTx emulation disabled, and MSI-X capability enabled. Bus mastering is also enabled on the device.
There is no MSI capability so it's not enabled.

I've just read a PCIe change notice that also says that each of the bridges on the way has to have bus master bit set as well. I am not doing it explicitly but not sure if the BIOS is not setting it up already?

Here's some code below for parts of the routines I'm using to set things up.

Enabling MSI-X

Code: Select all

  v = pci_read_dword(dev, msix_off);
  v |= 0x80000000;
  pci_write_dword(dev, msix_off, v);

Masking all vectors

Code: Select all

  uint32_t msix_bir = pci_read_dword(dev, dev->msix_off + 4);
  uint32_t msix_table_off = msix_bir & ~7;
  uint64_t tmp;

  msix_bir &= 7;

  dev->msix_sz = 1 + ((pci_read_dword(dev, dev->msix_off) >> 16) & 0x7FF);

  tmp = dev->bar[msix_bir + 1];
  tmp <<= 32;
  tmp |= dev->bar[msix_bir];
  tmp += msix_table_off;
  dev->msix = (uint32_t *)tmp;
  dev->msix_bitmap = zalloc_bitmap(dev->msix_sz);

  uint32_t pba_bir = pci_read_dword(dev, dev->msix_off + 8);
  uint32_t pba_table_off = msix_bir & ~7;

  pba_bir &= 7;

  tmp = dev->bar[pba_bir + 1];
  tmp <<= 32;
  tmp |= dev->bar[pba_bir];
  tmp += pba_table_off;
  dev->pba = (uint64_t *)tmp;

  /* Mask everything */
  uint32_t ctrl = pci_read_dword(dev, dev->msix_off);
  pci_write_dword(dev, dev->msix_off, ctrl | (1 << 30));

  uint32_t i;
  for (i = 0; i < dev->msix_sz; i++) {
    dev->msix[i * 4] = 0;
    dev->msix[i * 4 + 1] = 0;
    dev->msix[i * 4 + 2] = 0;
    dev->msix[i * 4 + 3] = 1;
  }

  /* unmask everything */
  pci_write_dword(dev, dev->msix_off, ctrl & ~(1 << 30));

Set up a single vector

Code: Select all

  uint32_t ctrl = pci_read_dword(dev, dev->msix_off);
  pci_write_dword(dev, dev->msix_off, ctrl | (1 << 30));

  dev->msix[msix_vec * 4] = MSI_TAG | (id << 12);
  dev->msix[msix_vec * 4 + 1] = 0;
  dev->msix[msix_vec * 4 + 2] = irq & 0xFF;
  dev->msix[msix_vec * 4 + 3] = 0;

  pci_write_dword(dev, dev->msix_off, ctrl & ~(1 << 30));

I'm getting most of my information on how to set this up from this document:
http://www.pcisig.com/specifications/co ... -x_ecn.pdf

Posted: **Sat May 05, 2012 8:53 am**

Sorry that I cannot be of more help, but I've only done MSI and not MSI-X yet.

I also have a network card that I cannot make generate interrupts. It is a RTL8169, and the driver works on several other PCs, but not on this particular one (an Intel Core Duo). This particular network card doesn't support neither MSI nor MSI-X, so I have to run it with interrupts. The IRQ number should be correct (verified with Windows XP, that works with the card). I've verfied that several interrupt sources are pending in the card, and also that PCI reports the IRQ as asserted, but the ISR is never called.

You should probably try to check with PCI if the interrupt is pending, and additionally with the network card that there are pending interrupts, and that interrupts are not masked.

OSDev.org

MSI-X delivery problem

MSI-X delivery problem

Re: MSI-X delivery problem

Re: MSI-X delivery problem

Re: MSI-X delivery problem