USB on Multiple Virtual Machines

[foliagecanine]

Hello.
So I've recently "finished" my xHCI controller driver, and realized that I had only been testing it in QEMU.
The next VMs I am testing it with are VMware and Virtualbox.
Both of these allow virtualizing real USB devices so I am not just testing the QEMU built-in ones. (I know that QEMU has virtualizing too, but I'm on WSL2 right now and it does not support USB devices. Maybe I'll boot up Linux sometime and try it.)

I also tried testing my OS on the only computer I'm willing to test it on that has xHCI support. Unfortunately, I could not get it to attach a device to the xHCI rather than the EHCI, so I just gave up. I even turned off the "legacy USB controller" in the BIOS. It initializes the xHCI controller all right, but detects no attached device on the single port that shows up (or its USB2 companion). The laptop has 4 USB ports physically, but only 1 port pair on the xHCI controller.

Here are the results:

Code: Select all

UHCI:
QEMU - Works
VMWare - Works (with real (virtualized) devices too)
VBox - Not supported by software

xHCI:
QEMU - Works
VMware - Fails
VBox - Fails

My main concern here is the xHCI. Neither VMware or VBox got a device to work fully.
In VMware, it manages to address the devices, but fails when trying to get the Device Descriptor (it times out).
VirtualBox fails even worse, as it fails when trying to enable a slot (it also times out).

I would like to be able to test with Bochs as well, but unfortunately I haven't been able to figure out how to get USB devices connected to the emulator.

Does anyone know why this is happening? Is there some quirk with QEMU that prevents it from working with the other two?
Anyone have experience with Bochs' USB?

Note: The serial logs from my OS for VMWare, VirtualBox, and QEMU are attatched.

[BenLunt]

The next VMs I am testing it with are VMware and Virtualbox.

This is where the frustration comes in. Works just fine on the current emulator (or hardware) and then move to another and it no longer works. Been there, done that.
The thing you have to look for is why is there a difference. What makes one emulator/hardware different than the next.

1) Version of hardware
2) BAR is 32-bit or 64-bit
3) Port Pairs are in different sequence (USB 3 before USB 2, or visa-versa)
4) xHCI uses 32-bit address registers or 64-bit address registers.
5) Context size is 32 or 64. 32 in one and 64 in the other.
6) Doorbell offset is at different address.
7) Maybe Runtime is too.
8) One has multiple scratchpads, one doesn't.
9) One allows port power control, one doesn't.
i.e.: Why is the hardware/emulation different than the one my code works with. Then go from there.

I would like to be able to test with Bochs as well, but unfortunately I haven't been able to figure out how to get USB devices connected to the emulator.
Anyone have experience with Bochs' USB?

I have had a "little" experience with Bochs and the xHCI emulation....

Code: Select all

/////////////////////////////////////////////////////////////////////////
// $Id: usb_xhci.cc 13519 2018-05-27 18:33:07Z sshwarts $
/////////////////////////////////////////////////////////////////////////
//
//  Copyright (C) 2010-2017  Benjamin D Lunt (fys [at] fysnet [dot] net)
//                2011-2018  The Bochs Project

ummm, sorry, I had to. It's late. My sense of humor goes unchecked. :-)

First, you have to add the xHCI to the PCI:

Code: Select all

pci: enabled=1, chipset=i440fx, slot1=cirrus, slot2=usb_xhci

Then, you have to include the xHCI:

Code: Select all

usb_xhci: enabled=1, port3=mouse

You can even indicate what speed you wish to use:

Code: Select all

usb_xhci: enabled=1, port1=disk:"hd.img", options1=speed:super

You can turn on debugging via the CONFIG button. All BX_DEBUG() lines will now be printed to the log file.

Back to your logs.

Note: The serial logs from my OS for VMWare, VirtualBox, and QEMU are attatched.

I looked at your VBoxLog.txt file. What is:

Code: Select all

0 1 1 FFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFF 0 0 0
FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFF
0 0 0 FFFFFFFFFFFFFFFF

At first glance, I would think these are reads from what you think is mem-mapped address space, but you have missed the address.

Code: Select all

[xHCI] Detected 14 ports
[xHCI] Found 8 USB 2 ports and 6 USB 3 ports.

I think you see it, but remember that there are only 8 physical sockets, all 8 support USB 2, only 6 of them support USB 3.

Code: Select all

[xHCI] Port 6: USB 2, Phys 6, PortPair 255
[xHCI] Port 7: USB 2, Phys 7, PortPair 255

The 255 does indicate that this port does not pair with a USB 3, correct?

You got a bit further with the VMWare emulation.

It takes time and testing. Test/Check things that you don't think you would need to check. It is usually something like that.

In the VBoxLog, are you reading from the correct address given from the BAR? It doesn't appear that you are, showing the FFFFF's.

Ben
- http://www.fysnet.net/the_universal_serial_bus.htm

[bloodline]

I would like to be able to test with Bochs as well, but unfortunately I haven't been able to figure out how to get USB devices connected to the emulator.
Anyone have experience with Bochs' USB?

I have had a "little" experience with Bochs and the xHCI emulation....

Code: Select all

/////////////////////////////////////////////////////////////////////////
// $Id: usb_xhci.cc 13519 2018-05-27 18:33:07Z sshwarts $
/////////////////////////////////////////////////////////////////////////
//
//  Copyright (C) 2010-2017  Benjamin D Lunt (fys [at] fysnet [dot] net)
//                2011-2018  The Bochs Project

ummm, sorry, I had to. It's late. My sense of humor goes unchecked.

Ben
- http://www.fysnet.net/the_universal_serial_bus.htm

Then I think I’m going to need your help when I get around to implementing USB in my project!!

But first I’ve got to figure out why I get random lockups in QEmu and not in VirtualBox

[foliagecanine]

I've looked through the list, and the only things I think would affect it are
6, 7,
8 (I don't use scratchpads, but maybe they can affect it somehow),
and 9 (but it shouldn't affect either one: VMware enables each device correctly, but fails when reading the device descriptor; VirtualBox fails when trying to enable a slot, which shouldn't care about port power).

What is: <hexdump>
At first glance, I would think these are reads from what you think is mem-mapped address space, but you have missed the address.

This is a hexdump of the HC OPS (I used it when testing real hardware, but I never took it out). I'm fairly certain it is at the correct BAR, since it resets the xHC fine. I think there's something wrong with my print function, since it should be a DWORD read/print:

Code: Select all

uint32_t *dump = (uint32_t *)xc->hcops;
for (uint8_t i = 0; i < 4; i++) {
	dbgprintf("%X %X %X %X\n", dump[0+(i*4)], dump[1+(i*4)], dump[2+(i*4)], dump[3+(i*4)]);
}

----------------------------------------------------------------------------------------------

I think you see it, but remember that there are only 8 physical sockets, all 8 support USB 2, only 6 of them support USB 3.

Code: Select all

[xHCI] Port 6: USB 2, Phys 6, PortPair 255
[xHCI] Port 7: USB 2, Phys 7, PortPair 255

The 255 does indicate that this port does not pair with a USB 3, correct?

Yes, 0xFF/255 does indicate that this port is unpaired. Therefore, my code correctly identifies 6 USB2/3 ports and 2 USB2-only ports, or 8 physical ports.

I'm going to start just changing things in the code until something happens. I'm probably going to start by waiting for a response before sending the status TRB, since that's a QEMU thing.

[foliagecanine]

I finally got around to trying it on Bochs, and it didn't work.

My OS's log is attached, but in short it gives a status error 4 when trying to set the device address.

Additionally, the (minified) Bochs VM log is attached. I am having trouble understanding what is going wrong.
(I removed tons of "useless" reads because the file size was too big for the forum)

These lines confuse me the most:

Code: Select all

00191692103d[XHCI  ] Dump command trb: 11(dec)  (0x000000000088a000 0x00000000 0x01002C01) (1)
00191692103d[XHCI  ]    slot_context->int_target = 0, slot_context->max_exit_latency = 0
00191692103d[XHCI  ]    ep_num = 1, speed = -1, ep_context->max_packet_size = 64
00191692103d[XHCI  ] Interrupt Fired.
00191692103d[XHCI  ] 0x0000000000835010: Command Ring: SetAddress TRB (bsr = 0) (addr = 2) (slot = 1) (returning 4)

Why is the "speed" listed as -1? Is that why it is returning 4?

[BenLunt]

My OS's log is attached, but in short it gives a status error 4 when trying to set the device address.

So if you go to line 2321 in the Bochs xHCI source, you will see the Address Device code.
A Status Error of 4 is a TRANSACTION_ERROR. This comes from the send_set_address() call returning an error.
Since the send_set_address() call is part of the hardware, i.e.: the hardware actually sends the set_address command, it will only fail if the host has not been setup correctly beforehand.
This most likely means that it is not your SET_ADDRESS function, but something before you get to this point.

These lines confuse me the most:

Code: Select all

00191692103d[XHCI  ] Dump command trb: 11(dec)  (0x000000000088a000 0x00000000 0x01002C01) (1)
00191692103d[XHCI  ]    slot_context->int_target = 0, slot_context->max_exit_latency = 0
00191692103d[XHCI  ]    ep_num = 1, speed = -1, ep_context->max_packet_size = 64
00191692103d[XHCI  ] Interrupt Fired.
00191692103d[XHCI  ] 0x0000000000835010: Command Ring: SetAddress TRB (bsr = 0) (addr = 2) (slot = 1) (returning 4)

Why is the "speed" listed as -1? Is that why it is returning 4?

The speed of -1 is used within the Bochs code to indicate to not check the speed when an evaluate_ep_context() call is made. (Line 2978) This is internal and has nothing to do with your code. The speed of the device may not have been determined yet, so we don't check it at this point. You may ignore the -1 at this point in time.

Hope that helps,
Ben

[foliagecanine]

One more thing. I forgot to post my bochsrc. I wanted to make sure I was configuring it correctly:

[BenLunt]

One more thing. I forgot to post my bochsrc. I wanted to make sure I was configuring it correctly:

I don't use plugins, so I can't comment on this line. However, I believe it to be identical in function.

Code: Select all

plugin_ctrl: unmapped=true, biosdev=true, speaker=true, extfpuirq=true, parallel=true, serial=true, gameport=true, iodebug=true, usb_xhci=true

This is correct.

Code: Select all

pci: enabled=1, chipset=i440fx, slot1=usb_xhci

Good.

Code: Select all

debug: action=ignore, usbdevctl=report, usb_xhci=report, usb_uhci=report, usb_hid=report

I would remove this line. It is not enabled, so it won't have any conflict with the hardware, though there might be some conflict with your software once you get it running if you ever set this to enabled.

Code: Select all

mouse: type=ps2, enabled=false, toggle=ctrl+mbutton

And these look correct, though the =none is implied so you can remove them if you wish.

Code: Select all

usb_xhci: enabled=1, port1=none, options1=none, port2=none, options2=none, port3="mouse", options3=none, port4=none, options4=none
usb_uhci: enabled=1, port1=none, options1=none, port2=none, options2=none

[foliagecanine]

I had a much simpler one, but when I saved the configuration after enabling logging (via the config menu) it added a bunch of lines.
Alright. So it has to be something wrong in my code...

Is there a way to get the GDB debugging symbols for Bochs without downloading and building it from source?

[foliagecanine]

Success!! I finally got Bochs to work!
I decided to download the source code and build it myself while adding a few more debug lines:

usb_xhci.cc:3038

Code: Select all

  BX_DEBUG(("Broadcast Packet"));
  printf("port: %d\n",port);

  if (BX_XHCI_THIS hub.usb_port[port].device != NULL)
    ret = BX_XHCI_THIS hub.usb_port[port].device->handle_packet(p);

  if (ret == USB_RET_NODEV)
	BX_DEBUG(("No device"));

I figured out that it was saying that there was no device attached to that port.
After a while, I found out that my port pairing code had an error in it. Rather than setting the rh_port_num to the port that it was configuring, it set it to the port PAIR.

Now it looks like it is functioning well in Bochs.
Unfortunately, it did not help the situation with VMware or VirtualBox.

[BenLunt]

Success!! I finally got Bochs to work!
I decided to download the source code and build it myself while adding a few more debug lines

Good for you. This is a good thing. You should be able to debug more thorough now.

After a while, I found out that my port pairing code had an error in it. Rather than setting the rh_port_num to the port that it was configuring, it set it to the port PAIR.

This comes back to my original list, number 3 in fact:

3) Port Pairs are in different sequence (USB 3 before USB 2, or visa-versa)

The port pairs were different, so you were getting a different result.

Again, sometimes it is a simple error, though figuring it out is the hard part.

I commend you for taking the effort to have a working build of Bochs. This will help you a lot.

Ben
- http://www.fysnet.net/osdesign_book_series.htm