Handling devices that only supports 32-bit addresses

[rdos]

Thought I'd get some feedback + other solutions for this. As many modern chips use memory-mapped IO, and most of them comes in both 32 bit and 64 bit versions, it seems like a relevant issue.

Here is how I handle them:

1. Network chips. I allocate static buffers for the buffer rings that always are 32-bit in the specific driver

2. HDA audio chips. Same. Allocate static sound buffers below 4G and copy data to them as part of the mixing process

3. USB. This is a tricky one. OHCI and UHCI have no support at all for 64-bit addressing. In EHCI it is optional, and you would want to pass pointers as is when the chip support 64-bit addressing. I solve this in the USB interface module by copying incoming requests to buffers that are 32-bit when the USB driver indicates it cannot handle 64-bit addresses, and the computer has any available address above 4G. This causes a lot of overhead for USB 1 and 1.1 devices that are not on hubs.

4. SATA discs. In AHCI, supporting 64-bit addressing is optional, and there are machines with 32-bit only AHCI controllers equipped with more than 4G of memory. I solve this in the low-level disc handler module by allocating all buffers below 4G when the disc driver indicates it cannot handle 64-bit addressing. This doesn't cause much overhead as no copying is required.

Feel free to post how you handle these issues in the thread.

[Brendan]

Hi,

Feel free to post how you handle these issues in the thread.

I always copy data to/from buffers that were allocated by the driver (and therefore never have to worry about "32-bit addressing only" hardware needing to access data at 64-bit physical addresses); because device drivers run as user-space processes (that communicate with other processes via. messaging) and aren't allowed to mess with the data in other process' virtual address spaces directly (e.g. using bus mastering or DMA). Note: This is slower.

For most OSs (e.g. Windows, Linux, etc; where user-space processes aren't protected from device drivers) I think they avoid the copying whenever possible:

• if the physical address for the data being transferred happens to be a 32-bit physical address already (or the device handles 64-bit addresses); then that address is used for the transfer "as is" with no copying
• if the data being transferred has a 64-bit physical address and the device doesn't handle 64-bit addresses; then either:
  • the kernel copies the data to/from temporary pages that do have 32-bit physical addresses; or
  • the kernel "permanently" replaces the physical page with a page that has a 32-bit physical address so that the device driver doesn't have to worry about it.

Note: If the kernel provides some sort of "get 32-bit physical address of virtual page" function, then this function could replace the page if/when needed, without the device driver knowing or caring if the original virtual page was replaced or not.


Cheers,

Brendan

[rdos]

if the physical address for the data being transferred happens to be a 32-bit physical address already (or the device handles 64-bit addresses); then that address is used for the transfer "as is" with no copying

That means you need to traverse the page-allocation chain for every buffer that might need relocation. I decided not to do this and instead check a global variable about the presence of any memory page above 4G.

the kernel "permanently" replaces the physical page with a page that has a 32-bit physical address so that the device driver doesn't have to worry about it.

I thought about this possibility in the USB-driver, but decided it didn't work properly because memory buffers passed were usually not page-aligned. This created big problems when page-table mappings for other objects where also changed, which would need CR3-shoot-down.

For USB, there is also a request object, which has a static buffer of a certain size, which is used by several USB devices. In this case, the buffer is forced to be in lower physical memory at allocation time when needed, and can be passed to the driver with no translation.

[Owen]

Most device driver interfaces have a "buffer" abstraction which all DMA is forced through. When the driver allocates the buffer, the system allocates memory accessible by the device (this might extend to e.g "bottom 16mb, physically contiguous and not crossing a 64kB boundary" at the bottom end of things where the interface in interacts with legacy ISA devices). If the buffer came from elsewhere, then it might either be remapped or a bounce buffer used when mapped for DMA (with copyback during the unmap operation for the bounce buffer case)

Also note that both modern AMD and Intel systems implement IOMMUs which can do this mapping for you, though often this feature needs enabling in the BIOS and is removed from lower model CPUs. On AGP systems some operating systems would use portion of the GART also, though the GART is of limited size.