OSDev.org

Not a bad overview of the history of PC expansion buses and the way current PCIe systems work. It specifically gives a decent (if whimsical) explanation of PCIe 'lanes'.

Unfortunately, one thing that isn't made clear is that each PCIe lane is an 8-bit connection; thus, a x16 slot can move sixteen one-byte packets at once, for a total of 128 bits per bus cycle. The reason this is relevant is because older bus interfaces often were discussed in terms of the number of bits transferred (e.g., ISA-8, ISA-16, PCI-32), rather than the number of bytes. This gets even ore confusing because PCIe is usually described as a 'serial' bus; this is because the physical lanes are in fact serial (one-bit) connections. However (IIUC), the logical connection system is arranged so that the lanes buffer and demux these serial signals into eight-bit packets before passing them to the peripheral hardware.

Actually, I want to ask about something I have been wondering: is anyone aware of any consumer-grade (i.e., not server) motherboards with x32 PCIe slots, and any video or networking devices in that connection form factor, and does anyone have any insight into why most remain x16 devices? I can imagine a number of possible reasons (cost, form factor size, lack of advantages in the wider connections, etc.), but I don't know which of them actually play a role in the limited adoption of the largest standard PCIe lane form factor.

I don't think any consumer-grade board has ever shipped with an x32 PCIe slot, and I can think of a few reasons for it:

• - Lack of available PCIe lanes: most consumer-grade processors and chipsets didn't actually have enough lanes to power an x32 slot (until very recent times... I don't know, maybe some Threadripper has more than 24 lanes?)
  - Lack of need for such high speeds: graphics cards are obviously ok with x16 slots, and even USB3.1 could fit into a couple of PCIe 1.0 lanes, so I can't think of many consumer applications for it.
  - Size: An x32 slot would occupy almost the whole length of a mATX board I reckon :O

So, mostly the reasons you already listed, basically.

I dont think there is much use in including 32x lanes. The only pcie devices that can potentially use more than 16x are high end graphics cards but they use that bridging to make it appear as one big-@$$ graphics card. I haven't tried it personally. CUDA development allows to detect multiple GPUs and load datas to each GPU selectively so there are lot of workaround. I believe older tools i.e. openMP, openCL perhaps allows to do that also.

For more compute and data intensive loads, there are also NVidia NVLINK bus that appears to leave PCIe speed in the dust. Perhaps some consumer motherboard will ship with that in the future. Servers are already shipping with NVLINK.

https://blogs.cisco.com/datacenter/cisc ... ity-for-ai