OSDev.org

Posted: **Sun Oct 18, 2020 11:23 pm**

Hello, I'm developing my xHCI drivers. I reset xHC by following the instructions of the Intel xHCI manual. An interrupt comes from xHCI on my laptop, but no interruption happens on QEMU. I suspect this is caused by my wrong configuration of the MSI-X table because the laptop has only MSI capability for xHCI, and QEMU has only MSI-X capability for it.

My full code: https://github.com/toku-sa-n/ramen/tree ... _from_qemu

This is the initialization of MSI-X. (kernel/src/device/pci/config/extended_capability/msi_x.rs)

Code: Select all

impl<'a> CapabilitySpec for MsiX<'a> {
    fn init_for_xhci(&self, config_type_spec: &TypeSpec) {
        let base_address = config_type_spec.base_address(self.bir());
        let mut table = self.table(base_address);

        let pending_base = config_type_spec.base_address(self.pending_bir());
        self.pending_bit_table(pending_base)[0] = 1;
        table[0].init_for_xhci();

        self.enable_interrupt();
    }
}

methods

Code: Select all

impl<'a> MsiX<'a> {
    fn enable_interrupt(&self) {
        let val = self.registers.get(self.base) | 0xf000_0000;
        self.registers.set(self.base, val);
    }
}
// kernel/src/device/pci/config/extended_capability/msi_x.rs
bitfield! {
    #[derive(Debug)]
    #[repr(transparent)]
    struct Element(u128);

    u32, from into MessageAddress, message_address,set_message_address: 31, 0;
    u32, from into MessageData, message_data, set_message_data: 95, 64;
    masked, set_mask: 96;
}
impl Element {
    fn init_for_xhci(&mut self) {
        self.message_address().init_for_xhci();
        self.message_data().init_for_xhci();
        self.set_mask(false);
    }
}

// kernel/src/device/pci/config/extended_capability/mod.rs
bitfield! {
    #[repr(transparent)]
    pub struct MessageAddress(u32);

    redirection_hint, set_redirection_hint: 3;
    u8, destination_id, set_destination_id: 19, 12;
    fixed_value, set_fixed_value: 31, 20;
}

impl MessageAddress {
    pub fn init_for_xhci(&mut self) {
        info!("LOCAL APIC ID: {}", Self::get_local_apic_id());
        self.set_destination_id(Self::get_local_apic_id());
        self.set_redirection_hint(false);
        self.set_fixed_value(0xfee);
    }

    fn get_local_apic_id() -> u8 {
        let accessor = single_object::Accessor::<u32>::new(LOCAL_APIC_ID_REGISTER_ADDR, 0);
        u8::try_from(*accessor >> 24).unwrap()
    }
}
bitfield! {
    #[repr(transparent)]
    pub struct MessageData(u32);

    vector, set_vector: 7, 0;
    delivery_mode, set_delivery_mode: 10, 8;
    level, set_level: 14;
    trigger_mode, set_trigger_mode: 15;
}

impl MessageData {
    pub fn init_for_xhci(&mut self) {
        self.set_level_trigger();
        self.set_vector(0x40);
        self.set_delivery_mode(0);
    }

    fn set_level_trigger(&mut self) {
        self.set_trigger_mode(true);
        self.set_level(true);
    }
}

Am I missing something?

Posted: **Mon Oct 19, 2020 6:35 pm**

tokusan wrote:Hello, I'm developing my xHCI drivers. I reset xHC by following the instructions of the Intel xHCI manual. An interrupt comes from xHCI on my laptop, but no interruption happens on QEMU. I suspect this is caused by my wrong configuration of the MSI-X table because the laptop has only MSI capability for xHCI, and QEMU has only MSI-X capability for it.

I don't read RUST that well, so I can't decipher your code.

Have you set bit 15 in the PCIe Configuration-->Extended Capabilities-->MSIX Configuration WORD to enable MSI-X? With this being said, did you disable MSI with bit 0 in the MSI Configuration WORD?

Also, the comment that QEMU "only" supports MSI-X may be wrong, though I don't think you meant to mean it that way. QEMU supports PIN based interrupts as well.

Ben
- http://www.fysnet.net/the_universal_serial_bus.htm

Posted: **Mon Oct 19, 2020 7:20 pm**

BenLunt wrote:
tokusan wrote:Hello, I'm developing my xHCI drivers. I reset xHC by following the instructions of the Intel xHCI manual. An interrupt comes from xHCI on my laptop, but no interruption happens on QEMU. I suspect this is caused by my wrong configuration of the MSI-X table because the laptop has only MSI capability for xHCI, and QEMU has only MSI-X capability for it.
I don't read RUST that well, so I can't decipher your code.

Have you set bit 15 in the PCIe Configuration-->Extended Capabilities-->MSIX Configuration WORD to enable MSI-X? With this being said, did you disable MSI with bit 0 in the MSI Configuration WORD?

Also, the comment that QEMU "only" supports MSI-X may be wrong, though I don't think you meant to mean it that way. QEMU supports PIN based interrupts as well.

Ben
- http://www.fysnet.net/the_universal_serial_bus.htm

Thanks for your reply. Yes, I set the enable bit of MSI-X message control register. I can't disable MSI because there is no MSI capability. PCI capability pointer points MSI-X capability and the next pointer of MSI-X cap is 0.

And yes, I meant "only" to say QEMU doesn't have MSI capability. I forgot about PIN based interruption.

Posted: **Tue Oct 20, 2020 2:18 am**

From qemu's debugprints and traces:

Code: Select all

[email protected]:usb_xhci_runtime_write off 0x0028, val 0x00000100
[email protected]:usb_xhci_runtime_write off 0x0030, val 0x10ce9000
[email protected]:usb_xhci_runtime_write off 0x0034, val 0x00000000
xhci: invalid value for ERSTSZ: 256
xhci: asserted controller error
...
[email protected]:usb_xhci_oper_read off 0x0004, ret 0x00001000  <- USBSTS_HCE

Posted: **Tue Oct 20, 2020 2:22 am**

linuxyne wrote:From qemu's debugprints and traces:

Code: Select all

[email protected]:usb_xhci_runtime_write off 0x0028, val 0x00000100
[email protected]:usb_xhci_runtime_write off 0x0030, val 0x10ce9000
[email protected]:usb_xhci_runtime_write off 0x0034, val 0x00000000
xhci: invalid value for ERSTSZ: 256
xhci: asserted controller error
...
[email protected]:usb_xhci_oper_read off 0x0004, ret 0x00001000  <- USBSTS_HCE

Thanks! I'll investigate this. BTW how did you print these debug info?

Posted: **Tue Oct 20, 2020 2:26 am**

tokusan wrote:BTW how did you print these debug info?

The lines prefixed with numerical info (most likely PID, etc.) can be enabled at the qemu monitor by the command

Code: Select all

trace-event usb_xhci* on

The other two lines (that begin with xhci:) requires rebuilding qemu with an additional -DDEBUG_XHCI entry into its cflags (or equivalently uncommenting the corresponding #define in the source).

Posted: **Tue Oct 20, 2020 6:26 am**

Thanks. I fixed the value of ERSTSZ, and it seems event ring TRB is generated (QEMU log says that). Still no interruption happens...

Posted: **Tue Oct 20, 2020 9:52 am**

- qemu ignores MMIO writes to PBA.
- The vectors are all masked in the msix table.

Edit:

Code: Select all

    fn enable_interrupt(&self) {
        let val = self.registers.get(self.base) | 0xf000_0000;
        self.registers.set(self.base, val);
    }

It seems the above is trying to enable msi-x, but it also sets a mask:

Code: Select all

#define  PCI_MSIX_FLAGS_MASKALL 0x4000  /* Mask all vectors for this function */
#define  PCI_MSIX_FLAGS_ENABLE  0x8000  /* MSI-X enable */

You may want to OR with 0x8000_0000 alone.

Posted: **Tue Oct 20, 2020 10:07 pm**

linuxyne wrote:- qemu ignores MMIO writes to PBA.
- The vectors are all masked in the msix table.

Edit:
Code: Select all
    fn enable_interrupt(&self) {
        let val = self.registers.get(self.base) | 0xf000_0000;
        self.registers.set(self.base, val);
    }
It seems the above is trying to enable msi-x, but it also sets a mask:
Code: Select all
#define  PCI_MSIX_FLAGS_MASKALL 0x4000  /* Mask all vectors for this function */
#define  PCI_MSIX_FLAGS_ENABLE  0x8000  /* MSI-X enable */
You may want to OR with 0x8000_0000 alone.

Thanks. I fixed it. Although no changes occur, I noticed that MSI-X table is in fact not set correctly. I may use bitfield library wrongly. I'll continue investigating.

Posted: **Tue Oct 20, 2020 10:13 pm**

The way you are trying to generate the interrupt depends on writing to the PBA, and since qemu ignores such writes, that method will not work.

The other fixes are necessary for proper functioning of the driver, but they alone will not be able to overcome the limitation that qemu ignores PBA writes.

Posted: **Wed Oct 21, 2020 3:05 am**

linuxyne wrote:The way you are trying to generate the interrupt depends on writing to the PBA, and since qemu ignores such writes, that method will not work.

The other fixes are necessary for proper functioning of the driver, but they alone will not be able to overcome the limitation that qemu ignores PBA writes.

I confirmed that we can't use PBA on QEMU, but then I can't understand why because there are lines related to PBA in QEMU's source code (inside hw/pci/msix.c). Anyway, I'll write codes which doesn't depend on the interruption from xHCI. Thank you for spending time for me.

Posted: **Wed Oct 21, 2020 3:46 am**

tokusan wrote:I confirmed that we can't use PBA on QEMU, but then I can't understand why because there are lines related to PBA in QEMU's source code (inside hw/pci/msix.c).

Probably because the driver/OS isn't expected to write to PBA; it is managed by the device.
[Edit: PBA is indeed supported by qemu; it just doesn't allow the guest to write to that memory area.]

Here is the patch that added a dummy handler for such writes to PBA.

If you are willing to test, you can change the function msix_pba_mmio_write to call msix_set_pending. That will allow the driver/OS to actually change the bit. [Edit: I tested this change in qemu with your latest fix_no_interrupt_from_qemu branch. The message "INFO: Interrupt from 0x40" is printed.]

Posted: **Wed Oct 21, 2020 9:13 am**

linuxyne wrote:If you are willing to test, you can change the function msix_pba_mmio_write to call msix_set_pending. That will allow the driver/OS to actually change the bit. [Edit: I tested this change in qemu with your latest fix_no_interrupt_from_qemu branch. The message "INFO: Interrupt from 0x40" is printed.]

Could you show me the edited msix_pba_mmio_write? I tried several codes, but couldn't get an interruption.

(BTW I said that event TRB was generated, but it turned out that this was not true. I misinterpreted QEMU's output. But at least the problem of the interruptions is solved (the way I want to implement is seemingly not supported by QEMU), so I marked this thread as solved.)

Posted: **Wed Oct 21, 2020 9:27 am**

tokusan wrote: Could you show me the edited msix_pba_mmio_write? I tried several codes, but couldn't get an interruption.

Below is a hack; it marks the vector# 0 as pending, regardless of the bit the driver/OS wants to
set in PBA. So, for testing, your code must always unmask vector# 0 (which it already does).

Code: Select all

static void msix_pba_mmio_write(void *opaque, hwaddr addr,
                                uint64_t val, unsigned size)
{
	PCIDevice *dev = opaque;
        msix_set_pending(dev, 0);
}

tokusan wrote: ...(the way I want to implement is seemingly not supported by QEMU),...

That's true, but I suppose that that way is only for testing. I do not think that commercial production OS and drivers actually write to PBA. They might read PBA to display interrupt status info. Moreover, if you add a usb-storage device to qemu's xhci bus, and run Linux on the guest, I can see that the xhci driver uses MSI-X, so the MSI-X functionality in qemu does work.

OSDev.org

[SOLVED]xHC interruption with MSI-X on QEMU doesn't work

[SOLVED]xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: xHC interruption with MSI-X on QEMU doesn't work

Re: [SOLVED]xHC interruption with MSI-X on QEMU doesn't work