Determining valid PCI address ranges

[Ethin]

So, I know that when the firmware initializes the hardware it assigns all the valid PCI device ranges and stuff (though I've no idea how it does this). By that, I mean that when you use the PCI I/O protocol in UEFI for example and you write to offset 0x00 of a devices BAR, the firmware knows if the write is a valid write or not and errors if its invalid. I'm wondering how I can do this within a kernel. Is this even possible or does each driver have to (manually) figure this stuff out on there own?

Another way to ask what I'm trying to say is: how would I go about determining the address range used by a given BAR (e.g.: if a device uses up to register offset 0x600, but only the datasheet says that, how does the firmware know that and is there a way I can tell that from something like ACPI)?

[qookie]

You can check how large the BAR is by checking how many address bits the device decodes from it. To do that, write all 1s into the BAR, then read the value back. Only the uppermost N bits will remain as 1s since they're what the device actually looks at to determine if it belongs to that BAR, and the remaining bits (which are all 0s) form the offset into the BAR, which you can use to calculate it's size. Do note that the BARs have some hardwired type bits which you need to take care of (by masking them off).

[Ethin]

You can check how large the BAR is by checking how many address bits the device decodes from it. To do that, write all 1s into the BAR, then read the value back. Only the uppermost N bits will remain as 1s since they're what the device actually looks at to determine if it belongs to that BAR, and the remaining bits (which are all 0s) form the offset into the BAR, which you can use to calculate it's size. Do note that the BARs have some hardwired type bits which you need to take care of (by masking them off).

So if I understand you right, I'd write all ones to the BAR, read back the value and ignore bits 2:1, and then count all leading zeros?

[Octocontrabass]

The number of trailing bits you need to ignore (treat as always 0) depends on the type of BAR. For memory BARs, it's four bits. For I/O BARs, it's two.

[Ethin]

The number of trailing bits you need to ignore (treat as always 0) depends on the type of BAR. For memory BARs, it's four bits. For I/O BARs, it's two.

Thanks. Would that be in Mbytes? Like if I count all the zeros, and there are only 4, would that be 4 MB? (Supposedly - according to the PCIe spec -- I could also check the PCIe capabilities structure for some of this info too. But that means the device has to be PCIe capable.)
Edit: nevermind, figured it out thanks to the OSDev wiki article -- thanks for the help!

[Ethin]

Okay... Final question and then I'll have figured this out. What do I do about 64-bit BARs? Do both BARs have the same size?

[Octocontrabass]

That question doesn't make sense. 64-bit BARs describe a single memory range, so there's one size. The only difference from a 32-bit BAR is that there are more address bits. The logic to find the size works exactly the same (but with more bits).

[Ethin]

That question doesn't make sense. 64-bit BARs describe a single memory range, so there's one size. The only difference from a 32-bit BAR is that there are more address bits. The logic to find the size works exactly the same (but with more bits).

I think I got it... My code is below. I only scan BARs that are nonzero,and I don't convert 64-bit base addresses into their full addresses because I was getting ridiculous results (e.g. one device claimed it needed 2 << 37 bytes of address space which is ridiculous). Just want to make sure this is correct:

Code: Select all

    if dev.header_type == 0x00 || dev.header_type == 0x01 {
        dev.bars
            .iter()
            .filter(|(idx, _)| **idx <= 5)
            .for_each(|(idx, bar)| {
                if *bar != 0x00 {
                    let i = idx;
                    let idx = match idx {
                        0 => BAR0,
                        1 => BAR1,
                        2 => BAR2,
                        3 => BAR3,
                        4 => BAR4,
                        5 => BAR5,
                        _ => 0,
                    };
                    let oldbar = read_dword(addr, idx);
                    write_dword(addr, idx, u32::MAX);
                    let bar = read_dword(addr, idx);
                    write_dword(addr, idx, oldbar);
                    let bar = if bar.get_bit(0) {
                        bar.get_bits(2..32)
                    } else {
                        bar.get_bits(4..32)
                    };
                    let bar = !bar;
                    info!("BAR {:X} consumes {} bytes", i, 2 << bar.count_ones());
                }
            });
    }

I ignore header type 2 because those addresses aren't directly mapped to the BAR registers like header types 0 and 1 are (though I should probably do those too...). This code gives me reasonable results:

0:0:0:0: Found Bridge (Host bridge) with vendor ID 8086 and device ID 29C0
0:0:1:0: Found Display controller (VGA compatible controller) with vendor ID 1234 and device ID 1111
BAR 0 consumes 33554432 bytes
BAR 2 consumes 8192 bytes
0:0:2:0: Found Network controller (Ethernet controller) with vendor ID 8086 and device ID 10D3
BAR 0 consumes 262144 bytes
BAR 1 consumes 262144 bytes
BAR 2 consumes 64 bytes
BAR 3 consumes 32768 bytes
0:0:3:0: Found Mass storage controller (Non-Volatile memory controller) with vendor ID 1B36 and device ID 10
BAR 0 consumes 32768 bytes
0:0:4:0: Found Mass storage controller (SATA controller) with vendor ID 8086 and device ID 2922
BAR 4 consumes 64 bytes
BAR 5 consumes 8192 bytes
0:0:5:0: Found Multimedia controller (Audio device) with vendor ID 8086 and device ID 293E
BAR 0 consumes 32768 bytes
0:0:1D:0: Found Serial bus controller (USB controller) with vendor ID 8086 and device ID 2934
BAR 4 consumes 64 bytes
0:0:1D:1: Found Serial bus controller (USB controller) with vendor ID 8086 and device ID 2935
BAR 4 consumes 64 bytes
0:0:1D:2: Found Serial bus controller (USB controller) with vendor ID 8086 and device ID 2936
BAR 4 consumes 64 bytes
0:0:1D:7: Found Serial bus controller (USB controller) with vendor ID 8086 and device ID 293A
BAR 0 consumes 8192 bytes
0:0:1F:0: Found Bridge (ISA bridge) with vendor ID 8086 and device ID 2918
0:0:1F:2: Found Mass storage controller (SATA controller) with vendor ID 8086 and device ID 2922
BAR 4 consumes 64 bytes
BAR 5 consumes 8192 bytes
0:0:1F:3: Found Serial bus controller (SMBus) with vendor ID 8086 and device ID 2930
BAR 4 consumes 128 bytes
PCIe scan complete; 13 devices found

The display controller has a bit of a large address space but I chalk that up to it being a graphics card. The only flaw with this code is that it scans addresses that are (supposed) to be 64-bits so I need to skip those if I already scan their lower dwords.

[Octocontrabass]

I don't convert 64-bit base addresses into their full addresses because I was getting ridiculous results

Sounds like you might have been putting the two dwords the wrong way around when assembling the whole qword.

Code: Select all

                    info!("BAR {:X} consumes {} bytes", i, 2 << bar.count_ones());

That should be a 1 instead of a 2. Right now, you're doubling all of the numbers you print.

The display controller has a bit of a large address space but I chalk that up to it being a graphics card.

It's actually pretty small compared to a typical graphics card.

[Ethin]

I don't convert 64-bit base addresses into their full addresses because I was getting ridiculous results

Sounds like you might have been putting the two dwords the wrong way around when assembling the whole qword.

I don't think so. I read the first BAR, then set bits 32..64 to the next BAR (BAR index + 4). Something like:

Code: Select all

let bar = read_dword(addr, idx) as u64; // idx contains current BAR index
if bar.get_bits(1 ..= 2) == 0x02 {
let _ = bar.set_bits(32 .. 64, read_dword(addr, idx + 4) as u64);
}
// normal size calculation code

(I also wrote all ones to both BARs -- that might've messed everything up too.)

Code: Select all

                    info!("BAR {:X} consumes {} bytes", i, 2 << bar.count_ones());

That should be a 1 instead of a 2. Right now, you're doubling all of the numbers you print.

Whoops?

The display controller has a bit of a large address space but I chalk that up to it being a graphics card.

It's actually pretty small compared to a typical graphics card.

Oh okay, good to know. If I had a secondary GPU I'd test with real-world hardware. Same if I had PCI passthrough. But I don't have any IOMMU groups and I don't really have any HW I could pass through anyway.

[Ethin]

For reference, my PCI code is here. The lines concerned are lines 286-341. Building the qwords gives me a shift-by-left overflow panic, so I'm really, really confused.

[nullplan]

You define "bar" as a dword by initializing it from "read_dword". I am not an expert on rust, but I'm going to assume variables work kind of like C++11's "auto", right? That would mean bar is a dword and shifting further than 32 bits does not work. But apparently you can specify a type explicitly, or so a short web search informed me.

[Ethin]

You define "bar" as a dword by initializing it from "read_dword". I am not an expert on rust, but I'm going to assume variables work kind of like C++11's "auto", right? That would mean bar is a dword and shifting further than 32 bits does not work. But apparently you can specify a type explicitly, or so a short web search informed me.

I type-cast that to a 64-bit integer, which results in zero-extension. I could just perform a qword read though... I presume PCIe supports that.

[Octocontrabass]

I type-cast that to a 64-bit integer, which results in zero-extension.

So, for BARs that only have 32 bits, did you continue to use those zero-filled bits in the later calculations? You're counting the number of bits that read back as zero after trying to set them to one, so that could give you an extra 32 zeroes.

[Ethin]

So I'm (starting) to get (almost right) results with this code:

Code: Select all

        let mut idx = 0;
        loop {
            if !dev.bars.contains_key(&idx) || dev.bars[&idx] == 0x00 {
                break;
            }
            let real_idx = match idx {
                0 => BAR0,
                1 => BAR1,
                2 => BAR2,
                3 => BAR3,
                4 => BAR4,
                5 => BAR5,
                _ => 0,
            };
            let oldbar = read_dword(addr, real_idx);
            let oldbar2 = if oldbar.get_bits(1..=2) == 0x02 {
                read_dword(addr, real_idx + 4)
            } else {
                0
            };
            write_dword(addr, real_idx, u32::MAX);
            if oldbar.get_bits(1..=2) == 0x02 {
                write_dword(addr, real_idx + 4, u32::MAX);
            }
            let mut bar = read_dword(addr, real_idx);
            let bar2 = if oldbar.get_bits(1..=2) == 0x02 {
                read_dword(addr, real_idx + 4)
            } else {
                0
            };
            write_dword(addr, real_idx, oldbar);
            if bar2 == 0x00 {
                write_dword(addr, real_idx + 4, oldbar2);
            }
            let ones = if bar2 == 0x00 {
                let bar = if bar.get_bit(0) {
                    bar.set_bits(0..2, 0)
                } else {
                    bar.set_bits(0..4, 0)
                };
                !bar.count_ones()
            } else {
                let mut bar = (bar2 as u64) << 32 | (bar as u64);
                if bar.get_bit(0) {
                    bar.set_bits(0..2, 0);
                } else {
                    bar.set_bits(0..4, 0);
                }
                !bar.count_ones()
            };
            info!("BAR {:X} consumes {} bytes", idx, 1 << ones.count_ones());
            if oldbar.get_bits(1..=2) == 0x02 {
                idx += 2;
            } else {
                idx += 1;
            }
        }

Doesn't quite work right though. Either skips BARs or tells me that a BAR needs 256M of RAM (which I don't think is right). Or it might be... Qemu args:

Code: Select all

qemu-system-x86_64 -drive format=raw,file=/home/ethin/source/kernel/target/x86_64-kernel-none/debug/boot-uefi-kernel.img -drive if=pflash,format=raw,file=OVMF.fd,readonly=on -drive file=disk-nvme.img,if=none,id=NVME01 -device nvme,drive=NVME01,serial=0001 -drive id=disk,file=disk-sata.img,if=none -device ahci,id=ahci -device ide-hd,drive=disk,bus=ahci.0 -usb -rtc clock=host -cpu host -smp cpus=8 -M q35 -name kernel -nographic -debugcon file:qemu.log -global isa-debugcon.iobase=0x402 -d int -D qemu2.log -device usb-audio,audiodev=alsa -audiodev alsa,id=alsa -device ich9-intel-hda -device hda-duplex,audiodev=alsa -no-shutdown -no-reboot -device qemu-xhci -device virtio-net,netdev=nic -netdev user,hostname=kernel,id=nic -device virtio-balloon -device virtio-serial-pci -device virtio-rng-pci,rng=rng0 -object rng-random,id=rng0,filename=/dev/urandom -device virtio-gpu --enable-kvm