USB on real hw

[Bonfra]

Some weeks ago I posted about making USB devices work on virtual machines, specifically, I was testing an MSD with an UHCI controller. After having it working I tinkered a bit with that code to make it work in a proper way and in a less prototype form. While it still is pretty bare bone and the specific UHCI controller implementation is still a bit clunky, since it works on QMEMU I thought I could make it work on real HW before making the code prettier and faster so that I always have a baseline to come back that I know work as I need.
Sadly I cannot communicate successfully with the device on real HW. the UHCI initialization phase appears to work as I'm able to enumerate the ports and find the attached device but when I try to send a control packet to set its address (so even before retrieving the default descriptor) the execution flow stops and I'm left waiting for the packets in the UHCI framelist to be processed.

Follow some parts of my code highlighting the process:

Code: Select all

// this functions expects a just resetted port to be present at address 0
void usb_register_device(usb_bus_t* bus)
{
    uint64_t addr = alloc_address(bus);

    // TODO: real hw hangs inside this function
    if(usb_set_address(bus, addr) != USB_TRANSFER_STATUS_OK)
        return;

    kernel_log("Done\n");

    // ....
}

usb_transfer_status_t usb_set_address(const usb_bus_t* bus, uint64_t addr)
{
    usb_request_packet_t setup;
    setup.type = USB_REQUEST_DIR_HOST_TO_DEVICE | USB_REQUEST_TYPE_STANDARD;
    setup.request = USB_REQUEST_SET_ADDRESS;
    setup.value = addr;
    setup.index = 0;
    setup.size = 0;

    return usb_transfer_control_out(bus, 0, 0, &setup);
}

usb_transfer_status_t usb_transfer_control_out(const usb_bus_t* bus, uint64_t addr, uint64_t endpoint, void* setup)
{
    usb_packet_t packets[2];
    memset(packets, 0, sizeof(usb_packet_t) * 2);

    packets[0].type = USB_PACKET_TYPE_SETUP;
    packets[0].maxlen = 8;
    packets[0].buffer = setup;
    
    packets[1].type = USB_PACKET_TYPE_IN;
    packets[1].maxlen = 0x800;
    packets[1].buffer = NULL;
    packets[1].toggle = 1;

    return uhci_transfer_packets(bus->hci.data, addr, endpoint, packets, 2);
}

usb_transfer_status_t transfer_packets(void* data, uint64_t addr, uint64_t endpoint, const usb_packet_t* packets, size_t num_packets)
{
    uhci_controller_t* controller = data;

    volatile alignas(0x20) transfer_descriptor_t tds[num_packets];
    for(size_t i = 0; i < num_packets; i++)
    {
        tds[i].link = i == num_packets - 1 ? TD_TERMINATE : ((uint32_t)(uint64_t)&tds[i + 1] | TD_DEPTH_FIRST);
        tds[i].flags = TD_STATUS_ACTIVE;

        if(i == num_packets - 1)
            tds[i].flags |= TD_IOC;

        if(port_status(controller, addr) == USB_PORT_STATUS_CONNECT_LOW_SPEED)
            tds[i].flags |= TD_LOW_SPEED;
        
        tds[i].maxlen = ((packets[i].maxlen ? packets[i].maxlen - 1 : 0) << 21) | (endpoint << 15) | (addr << 8);
        
        switch (packets[i].type)
        {
        case USB_PACKET_TYPE_SETUP: tds[i].maxlen |= TD_PID_SETUP; break;
        case USB_PACKET_TYPE_IN: tds[i].maxlen |= TD_PID_IN; break;
        case USB_PACKET_TYPE_OUT: tds[i].maxlen |= TD_PID_OUT; break;
        }

        if(packets[i].toggle)
            tds[i].maxlen |= TD_DATA_TOGGLE;

        tds[i].bufptr = (uint32_t)(uint64_t)packets[i].buffer;
    }

    volatile alignas(0x20) queue_head_t qh;
    qh.head_link = QH_TERMINATE;
    qh.element_link = (uint32_t)(uint64_t)&tds[0];

    for(uint32_t i = 0; i < 1024; ++i)
        controller->frame_list[i] = FRAMELIST_TERMINATE;
    controller->frame_list[0] = (uint32_t)(uint64_t)&qh | FRAMELIST_QH;

    sched_run(controller);

    while(!(inportw(controller->io + IO_USBSTS) & USBSTS_INT))
        asm("pause");

    sched_stop(controller);

    return USB_TRANSFER_STATUS_OK;
} 

Yes, the code still sucks and is pretty much equal to my last topic about USB stuff, but believe me a lot of structural rewriting happened somewhere else that doesn't impact the general behavior of this specific portion of code.
Here and here the parts of my repo that contains a less simplified version of the code showed above.
At the moment I'm trying to retrieve various dumps of the data that gets sent in real HW but it's a bit of a slow process not having copy-paste and a really small screen. It's gonna take a while so for the time being I'm posting just this and I'll add in a subsequent post the dumps.

PS:
Since my OS wasn't designed with BOCHS in mind I was only able to test this thing on QEMU, being the only VM I know that emulates UHCI. If you happen to know any other VM that also emulates it please let me know as it would be interesting to know how my code behaves there

[BenLunt]

Hi,

With real hardware, you need to make sure you write your packets to physical memory in the correct order, as well as flushing to physical memory. For example, make sure that all data that is pointed to by a TD has already been written to memory before you write the TD to memory. Make sure that all TDs are in memory before you write a Queue Head to memory that might point to any of these TDs. i.e. make sure that all associated physical memory and all pointers to this memory are valid and are written in the correct order (Buffers before TDs before Queue Heads before Frame Pointers).

Also, timing is an issue with real hardware. You must wait for the controller to be ready where as an emulator usually is always ready. One common mistake I have found is an implementation, after setting the port's enable bit, won't wait for that enable bit to become set before trying to execute TDs. In an emulator, when you write a 1 to the Enable bit, it is (usually instantly) set. In real hardware, writing a 1 to the Enable bit will usually wait 10us to 100ms before it actually becomes set. The USBIF gives a set time you are to wait for this action. (You mentioned you have my book. I have a page or two on simply this fact about resetting UHCI ports (and their eccentricities). Have a look to make sure you are doing it correctly.)

Since my OS wasn't designed with BOCHS in mind I was only able to test this thing on QEMU, being the only VM I know that emulates UHCI. If you happen to know any other VM that also emulates it please let me know as it would be interesting to know how my code behaves there

In theory, any OS written "with QEMU in mind" should run just the same on Bochs, or any other emulator--in theory. Have you tried Bochs?

As long as you enable PCI, the UHCI is included. You can get a ready-made executable from https://bochs.sourceforge.io/getcurrent.html and use a bochrc.txt line like:

Code: Select all

pci: enabled=1, chipset=i440fx
usb_uhci: enabled=1, port1=disk:"hd.img"

See https://sourceforge.net/p/bochs/code/HE ... s/.bochsrc for an example bochsrc.txt file. Once Bochs has started booting, use the "Config" button to configure the USB's debugging capabilities. You can tell it to write all BX_DEBUG() messages to the log file. If you have the capabilities to compile Bochs (and I truly suggest you do), you can manipulate the interface to print all kinds of debug strings.

https://www.virtualbox.org/ also supports USB and is open source as well.

These are the three major emulators that I use. There are others, but I don't remember if they support USB or not.

Ben
- https://www.fysnet.net/the_universal_serial_bus.htm

[Bonfra]

you need to make sure you write your packets to physical memory in the correct order, as well as flushing to physical memory.

What do you mean here? if I instantiate some data structure isn't it always stored somewhere in memory? is there some buffer that holds RAM? Some caching thing I'm not aware of?

One common mistake I have found is an implementation, after setting the port's enable bit, won't wait for that enable bit to become set before trying to execute TDs.

Here I'm following the exact approach you suggest in your book:

Code: Select all

bool reset_port(uhci_controller_t* controller, uint64_t port)
{
    if(port >= controller->num_ports)
        return false;
    uint16_t addr = controller->io + IO_PORTSC + port * 2;

    // issue port reset
    outportw(addr, inportw(addr) | PORTSC_PORT_RESET);
    pit_prepare_one_shot(50);
    pit_perform_one_shot();
    outportw(addr, inportw(addr) & ~PORTSC_PORT_RESET);

    // wait for port to be ready
    for(uint8_t i = 0; i < 10; i++)
    {
        pit_prepare_one_shot(10);
        pit_perform_one_shot();

        uint16_t status = inportw(addr);

        if(!(status & PORTSC_CONNECT_STATUS))
            break;

        if(status & (PORTSC_STATUS_CHANGE | PORTSC_ENABLE_CHANGE))
        {
            // clear status change
            outportw(addr, status | PORTSC_STATUS_CHANGE | PORTSC_ENABLE_CHANGE);
            continue;
        }

        if(status & PORTSC_ENABLE)
            return true;

        outportw(addr, status | PORTSC_ENABLE);
    }

    return false;
}

And in real HW this code returns true so this should work fine.

Have you tried Bochs?

It's a bit too advanced for me, meaning that I may be able to start from a minimal working config file and morph it to my needs (specific display size, ahci stuff, uhci ...) but I can't forge it from scratch even looking at the general enormous template

https://www.virtualbox.org/ also supports USB and is open source as well.

Yes but regarding USB 1.1 only OHCI is supported

[rdos]

Regarding memory (and particularly PCI BARs), they need to be set not to cache contents to work properly.

Another problem with older USB standards is BIOS hand-off. Some real hardware will require this before you can successfully use them.

A further problem is that UHCI controllers often are companion controllers for EHCI, which means you need to hand over control to the ports from EHCI (unless this is done by default).

Timing is also important, and checking bits when the spec says so. As already pointed out, emulators doesn't handle this like real hardware.

IRQs are a particular problem for UHCI and OHCI. These typically won't support MSI or MSI-X, and sometimes ACPI doesn't deliver the correct IRQ #. I ended up not using IRQs for UHCI or OHCI, rather I use a 1 ms timer to poll hardware.

[Octocontrabass]

It's a bit too advanced for me, meaning that I may be able to start from a minimal working config file and morph it to my needs (specific display size, ahci stuff, uhci ...) but I can't forge it from scratch even looking at the general enormous template

On Windows, Bochs has a convenient GUI for setting up the basics; you can then manually modify any settings that the GUI doesn't know how to set appropriately.

Bochs doesn't support AHCI. If you need disk access, you might consider writing an IDE driver. IDE is much simpler than AHCI, so it shouldn't take very long. (IDE does have some rough edges though.)

[Bonfra]

Regarding memory (and particularly PCI BARs), they need to be set not to cache contents to work properly.

I did some research and it appears that in x86 it should just "work" as expected, but if after some other debugging nothing else pops out I'll add something about it in my paging driver.

Another problem with older USB standards is BIOS hand-off. Some real hardware will require this before you can successfully use them.

Yeah about that I couldn't find anything about it regarding UHCI, not in the specs nor in any piece of code I can find. I'm not sure about this but I also don't think is mentioned in Ben's book (or at least I cant find it referred to UHCI)

A further problem is that UHCI controllers often are companion controllers for EHCI, which means you need to hand over control to the ports from EHCI (unless this is done by default).

Shouldn't the BIOS setup everything for non-EHCI aware software? like auto-enable the hand over and stuffs? Anyway yes, my testing machine (an optiplex 780) has this EHCI-UHCI companion things

[BenLunt]

Hi,

rdos made some good points.

Another problem with older USB standards is BIOS hand-off. Some real hardware will require this before you can successfully use them.

Yeah about that I couldn't find anything about it regarding UHCI, not in the specs nor in any piece of code I can find. I'm not sure about this but I also don't think is mentioned in Ben's book (or at least I cant find it referred to UHCI)

Appendix M shows the Legacy support for all four controllers.

A further problem is that UHCI controllers often are companion controllers for EHCI, which means you need to hand over control to the ports from EHCI (unless this is done by default).

Shouldn't the BIOS setup everything for non-EHCI aware software? like auto-enable the hand over and stuffs? Anyway yes, my testing machine (an optiplex 780) has this EHCI-UHCI companion things

In theory, yes the BIOS should have set the EHCI controller to bypass and use the companion controllers. However, this doesn't mean that someone hasn't changed this setting. You should check the settings to be sure. If you have to change the EHCI yourself, you will need to read up on how the EHCI handles the companion controllers, which means you will need to detect the EHCI and parse its companion controller map. Probably something you are not wanting to do at the moment.

Octocontrabass makes a good point too. Bochs makes it quite simple to create a bochsrc.txt file. Either way, you really only need a few lines. Here is the one I used to boot your image file we talked about in the last thread.

Code: Select all

config_interface: win32config

romimage: file=C:/bochs/bochs/bios/BIOS-bochs-latest

cpu: model=broadwell_ult
cpu: count=1, ips=50000000, reset_on_triple_fault=1, ignore_bad_msrs=1
cpu: cpuid_limit_winnt=0

clock: sync=none, time0=local

memory: guest=512, host=512

vgaromimage: file=C:/bochs/bochs/bios/VGABIOS-lgpl-latest
vga: extension=vbe, update_freq=5, realtime=1, ddc=builtin

ata0: enabled=1, ioaddr1=0x1f0, ioaddr2=0x3f0, irq=14

ata0-master: type=disk, path="BonsOS.img", model="BonsOS.img", sect_size=512

boot: disk

log: log.txt

panic: action=ask
error: action=report
info: action=report
debug: action=ignore

mouse: enabled=0, type=imps2
private_colormap: enabled=0

pci: enabled=1, chipset=i440fx
magic_break: enabled=1

port_e9_hack: enabled=1

print_timestamps: enabled=0

com1: enabled=1, mode=file, dev=serial1.txt

usb_uhci: enabled=1, port1=disk:"..\FAT.IMG"

The first line is the line necessary to use the GUI part of the configuration I mentioned (clicking on the "Config" menu bar items).

To create a bochsrc file, simply run bochs with no parameters.

Code: Select all

bochs

It will allow you to set configuration for devices, then save to a file. Once you have a bochsrc.txt file, you can then skip the configuration with the 'q' parameter:

Code: Select all

bochs -q

Also note that Octocontrabass mentioned Bochs doesn't support AHCI. The firmware will load your kernel anyway, doing so as if it was IDE. However, his point is that once your kernel gets to the point where you want to read from the disk, you won't be able to anymore since you assume AHCI and Bochs is IDE only.

https://www.virtualbox.org/ also supports USB and is open source as well.

Yes but regarding USB 1.1 only OHCI is supported

Sorry, I had forgotten that it only supports OHCI, EHCI, and xHCI.

Ben

[Bonfra]

Appendix M shows the Legacy support for all four controllers.

It appears that my copy is missing this appendix; it jumps from K to Q. Maybe I have to download it from somewhere else?
Btw isn't legacy only about keyboard and mouse on USB controllers?

Thanks for posting the bochs config file, I got it finally booting on bochs, and (sadly) my USB driver works fine there. But now I can try to better debug the code looking for some quirckyness

EDIT
I erroneously edited this post deleting all of its content. I tried to restore the original meaning but surely the words have changed :(

[BenLunt]

Sorry, you must have an earlier edition.

Starting on page 38 of the UHCI specs, it describes the Keyboard and Mouse Legacy Support, then on page 39, the register interface to disable Legacy support.

A simple write to PCI configuration word 0xC0 of 0x8F00 does the trick.

Code: Select all

  pci_write_word(pci->bus, pci->dev, pci->func, 0xC0, 0x8F00);

Ben

[Bonfra]

Btw isn't legacy only about keyboard and mouse on USB controllers?

Yup even after adding legacy support for my driver it keeps on stalling on real hw. I tested on two different computers so now, more than before, I believe is more of a problem with my code rather than a hardware configuration issue

[Bonfra]

At this point, I'm not sure it's worth it to keep on fighting with this non-working code. I mean it may very well be the fact that the EHCI controller didn't perform automatically the hand over in both of my testing physical computers. do you think I should try implementing an EHCI driver, make that thing work, and later come back to UCHI to test if performing the hand over makes the thing work? If after implementing a whole new controller it still doesn't work I may cry XD

[BenLunt]

Since I have said it more than once, please note that when I say this again, it is of my own humble opinion, but...I would completely skip the EHCI altogether. It has so many idiosyncrasy's that you would be fighting it way more than you will the UHCI.

However, with that being said, you would first need to turn of legacy support (BIOS release ownership) for the EHCI.

Find the LEGACY Capability register in the EHCI's Extended Capabilities area (pointed to by bits 15:8 in the HCCPARAMS register (or EECP)), then set bit 24. The controller will clear bit 16 when the release has been made. You will need a timer and watch that both bit 24 has actually been set, and bit 16 has actually become clear.

Just to do that takes detecting the EHCI on the PCI(e) bus, setting up the PCI(e) registers for access, possibly remapping if necessary, as well as parsing the controllers EHCI's Extended Capabilities. Quite a bit of work. However, read a later note about the above procedure :-)

Now, once you have done that, you can write a zero to bit 0 of the ConfigFlag Register (base + 40h) telling the EHCI to release control to the companion controllers. However, some controllers will assume you have already setup the other registers as well, including the run bit.

Therefore, to do it correctly, and to answer your question, yes, you would have to have a (very minimal) working EHCI driver simply to release control to the companion controllers to continue with your UHCI.

Also, remember, EHCI will not see a connection of a full- or low-speed device. It will pass any full- or low-speed device off to these companion controllers anyway, so creating an EHCI driver will not help with USB mouse and USB keyboard support. Most thumb drives, though, are high-speed and the EHCI will allow those.

Back to my note about releasing ownership of the EHCI. I found at least one motherboard, it having multiple EHCI controllers on it, that if I released ownership of a higher PCI-numbered EHCI before the lower numbered EHCI, the BIOS would brick all other EHCI controllers on the motherboard. A complete hard reset was needed to regain control. This is from complete memory off the top of my head, so don't quote me on it, but I think it was the SMI on an ICH8 mobo. The ICH9 was okay. But that's from memory, so I could be remembering wrong.

I recommend that you stick with your UHCI code. Detect a connection on a port and do a reset. Wait the required interval and read the value back, printing to the screen. Then set the enable bit, again waiting the required interval, then read it back, printing to the screen. Check that the port indeed was enabled. Then move on to the next part, doing something and printing the result to the screen until you find something not right. Takes time, but once you find it, all that work seems to all of a sudden, become worth it.

I think I have said this before too..."Isn't USB fun?" Said with a bit of sarcasm :-)

Ben

[thewrongchristian]

Since I have said it more than once, please note that when I say this again, it is of my own humble opinion, but...I would completely skip the EHCI altogether. It has so many idiosyncrasy's that you would be fighting it way more than you will the UHCI.

Alas, to make USB2 based storage practical (I'm aiming for USB storage as well as USB kbd/mouse) EHCI is the minimum, and so many still useful machines are based on it that I for one intend to master this beast.

I think I have said this before too..."Isn't USB fun?" Said with a bit of sarcasm

I love USB.

I hate USB.

Does that about sum it up?

All joking aside, my actual USB keyboard and BBB MSD drivers are tiny, currently less than 200 and 300 lines respectively.

[Bonfra]

Since I have said it more than once, please note that when I say this again, it is of my own humble opinion, but...I would completely skip the EHCI altogether. It has so many idiosyncrasy's that you would be fighting it way more than you will the UHCI.

If I understood correctly you are suggesting to directly implement xHCI since it's backward compatible with older devices. While this is very nice my goals are still to target real hardware and the machines I use for testing at the moment don't have that specific controller. It will be hard and I'm going to spit blood trying to understand it but since I'm not on any deadline it's a risk I'm willing to make.

I recommend that you stick with your UHCI code. Detect a connection on a port and do a reset. Wait the required interval and read the value back, printing to the screen. Then set the enable bit, again waiting the required interval, then read it back, printing to the screen. Check that the port indeed was enabled. Then move on to the next part, doing something and printing the result to the screen until you find something not right. Takes time, but once you find it, all that work seems to all of a sudden, become worth it.
Ben

Yea after seeing all the steps you posted above just to be able to hand over the devices I feel kinda obligated to triple/quadruple check that absolutely nothing is wrong with the UHCI driver. Yea it's gonna take time but it's worth it. Thanks for the advice I'll post here the results

[Bonfra]

No updates for a few days sorry for my absence.
I've been busy turning this code inside out looking for any possible difference in behavior between real hw and emulation but it appears to be all the same.
Everything seems to be in the correct place with the right values until it gets to the critical part where real hw fails and emulation goes on.

Here is a side-by-side comparison of what I'm getting in QEMU and in real hw (QEMU on the right)


I also tried logging every single register of the UHCI controller at each point (even if lot of checks are present that would not make the execution continue if something is out of place) and they all match.

To be clear the part that it gets stuck is while waiting for the INT bit int the USBSTS register of the uhci controller to be set.

Code: Select all

while(!(inportw(controller->io + IO_USBSTS) & USBSTS_INT))
        asm("pause");

This is how (for the time being) I detect that all TDs had been executed, always resetting the schedule from the first one.