PCI BAR size detection returns completely bogus values

[mariuszp]

I'm trying to initialize BARs on real hardware, and i'm using this code:

Code: Select all

void pcidev_init(PCIDevice *dev) {
    *pcidev_put = dev;
    pcidev_put = &dev->next;

    u32 cmd_status = pci_config_read(dev->bus, dev->slot, dev->func, PCI_CONFIG_COMMAND_STATUS);
    pci_config_write(dev->bus, dev->slot, dev->func, PCI_CONFIG_COMMAND_STATUS, cmd_status & ~3); // disable I/O

    for (u32 i = 0; i < 6; i++) {
        PCIBar *bar = &dev->bars[i];

        u32 bar_low = pci_config_read(dev->bus, dev->slot, dev->func, PCI_CONFIG_BAR0 + 4 * i);
        if (bar_low == 0 || bar_low == U32_MAX) {
            bar->type = PCIBAR_TYPE_UNUSED;
            continue;
        }

        bool is_portio = !!(bar_low & 1);
        bool is_64bit = (bar_low & 7) == 4;

        if (is_64bit) {
            bar->flags |= PCIBAR_MEMORY_64BIT;
        }

        u32 bar_high = is_64bit ?
            pci_config_read(dev->bus, dev->slot, dev->func, PCI_CONFIG_BAR0 + 4 * i + 4) :
            0;
        
        pci_config_write(dev->bus, dev->slot, dev->func, PCI_CONFIG_BAR0 + 4 * i, U32_MAX);
        u32 size_mask = pci_config_read(dev->bus, dev->slot, dev->func, PCI_CONFIG_BAR0 + 4 * i);
        pci_config_write(dev->bus, dev->slot, dev->func, PCI_CONFIG_BAR0 + 4 * i, bar_low);

        printf("BAR %d: 0x%016lx\n", i, (bar_high << 32) | bar_low);

        if (is_portio) {
            bar->type = PCIBAR_TYPE_PORTIO;
            bar->base = bar_low & 0xFFFFFFFC;
            bar->size = (~(size_mask & 0xFFFFFFFC) + 1) & U32_MAX;
        } else {
            bar->type = PCIBAR_TYPE_MEMORY;
            bar->base = bar_low & 0xFFFFFFF0;
            bar->size = (~(size_mask & 0xFFFFFFF0) + 1) & U32_MAX;

            if (is_64bit) {
                bar->base |= (u64)bar_high << 32;

                pci_config_write(dev->bus, dev->slot, dev->func, PCI_CONFIG_BAR0 + 4 * i + 4, U32_MAX);

                u32 size_mask_high = pci_config_read(dev->bus, dev->slot, dev->func, PCI_CONFIG_BAR0 + 4 * i + 4);
                pci_config_write(dev->bus, dev->slot, dev->func, PCI_CONFIG_BAR0 + 4 * i + 4, bar_high);
                printf("size_mask = 0x%08lX, size_mask_high = 0x%08lX\n", size_mask, size_mask_high);
                u64 full_size_mask = ((u64)size_mask_high << 32) | (size_mask & 0xFFFFFFF0);
                bar->size = ~full_size_mask + 1;
            }

            printf("Mapping 0x%016lX, size=0x%016lX\n", bar->base, bar->size);
            mem_config((void*) bar->base, bar->size, MEM_UNCACHED);
        }

        if (is_64bit) {
            i++;
        }
    }

    // Restore the original status byte.
    pci_config_write(dev->bus, dev->slot, dev->func, PCI_CONFIG_COMMAND_STATUS, cmd_status);
}

(This is a slightly modified dialect of C, so i have types like u32, u64 etc)

In QEMU there are no 64-bit BARs, and on the laptop, the first BAR that's actually used by any device happens to be 64-bit, so i don't know if it's specific to this issue.

I'm detecting the device 8086:1903, with class 0x11, subclass 0x80, ProgIF 0x00.

The actual address reads correct and agrees with Linux: 0xef240000 (The low BAR value is 0xef240004 - non-prefetchable, 64-bit memory BAR). Linux says the size of this BAR is 32K.

But my code returns:

size_mask=0x80000004, size_mask_high=0x80002014.

this is clearly bogus, and gives me a massive BAR size.

I've been trying to figure out what i'm doing wrong but my code seems to do the same as linux: write all-1s to the low bar, read the mask, then write all-1s to the high bar, and read the mask. the mask comes out as this bogus value. I'm been looking at this for a bit and making no progress.

Can anyone see what might be wrong here?

[robfinch]

<not knowing too much about PCI>
Are interrupts / other devices locked out while doing this?
Why is the bar->size negated? Are the raw values of size_mask negative?

[mariuszp]

No PCI devices have been used at this point as they are still being enumerated and i'm probing for the sizes.

Yeah, the values returned after writing all-1s are the inverse of the highest address/port. so we negative them and add 1. It seems to match when linux is doing as well in pci/probe.c.

It works in QEMU but not on the laptop.

[robfinch]

It almost looks like the high and low halves of the 64-bit value are swapped. Could there be a read of the wrong half value? For example read of register 1 and 2, instead of 2 and 3? I see the loop increments the register by 1 and adjusts for 64-bit later. Can there be a mix of bar sizes?
How old is the laptop? Does it support 64-bit devices?

[mariuszp]

The laptop is recent but i don't know what model it is; it was running windows 10, and my employer let me keep it from work. Linux works on it. It does support 64-bit devices since the built-in chips report 64-bit BARs.

I don't think the halves are swapped; each half of the *mask* has bit 31 set, as well as a few random bits in the top half. But the mask should be 0 or more bits on top, followed by all ones on the bottom. so the mask is not even in a valid format. which is really strange, and again linux seems to detect them all just fine.

note that the printf() prints each part of 32-bit mask *separately*, so even if i have any issues with combining them, they are already wrong.

[mariuszp]

After all the debugging of this function, i found the real issue: pci_config_write() had a bug. It was writing the address into the data field, instead of the value. After fixing this, everything works perfectly. Thanks for your help @robfinch