Serial port output cannot be observed on VMware

[samseaborn]

I am attempting to add debug logging via the COM1 serial port into my microkernel, which is currently being executed parallel to another operating system (a Windows 11 instance) both as guests in a VMware instance. Below is the code I've been using to setup the COM1 serial port, along with an example of logging a simple debug string to the port. I can't seem to observe this output--even in isolated cases where my microkernel is not fully initialized parallel to the operating system executing it.

Code: Select all

__outbyte( 0x03F8 + 3, 0x80 ); // Enable DLAB to change the BAUD rate

__outbyte( 0x03F8 + 0, 1 ); // Set the BAUD rate to 115200
__outbyte( 0x03F8 + 1, 0 ); // Set the BAUD rate to 115200

__outbyte( 0x03F8 + 3, 3 ); // Disable DLAB, specify 8-bit data unit sizes, one stop bit, and no parity

__outbyte( 0x03F8 + 1, 0 ); // Disable interrupts

__outbyte( 0x03F8 + 2, 7 ); // Enable FIFO, clear out the receive and transmit buffers

Code for my output byte function + output byte string function, which was supplemented by derivatives provided by the host operating system; so I don't believe the error is here.

Code: Select all

void
output_byte(
    _input u16 port_address,
    _input u8  value 
    );

void
output_byte_string(
    _input u16   port_address,
    _input char* str,
    _input u32   len
    );

Code: Select all

output_byte PROC
    mov al, dl
    mov dx, cx
    out dx, al
    ret
output_byte ENDP
    
output_byte_string PROC
    push rsi

    mov rsi, rdx
    mov dx, cx
    mov ecx, r8d
    rep outsb dx, byte ptr [rsi]

    pop rsi
    ret
output_byte_string ENDP

I can't seem to observe any sort of serial port output produced given the configuration above. Before anybody asks, I did attempt to implement the entire example port setup from the OSDev wiki (code included below), which also goes on to test the port output by reading in loopback mode, and I can confirm I receive the 0xAE test byte when executing an IN instruction.

Code: Select all

    __outbyte( 0x03F8 + 1, 0x00 );    // Disable all interrupts
    __outbyte( 0x03F8 + 3, 0x80 );    // Enable DLAB (set baud rate divisor)
    __outbyte( 0x03F8 + 0, 0x03 );    // Set divisor to 3 (lo byte) 38400 baud
    __outbyte( 0x03F8 + 1, 0x00 );    //                  (hi byte)
    __outbyte( 0x03F8 + 3, 0x03 );    // 8 bits, no parity, one stop bit
    __outbyte( 0x03F8 + 2, 0xC7 );    // Enable FIFO, clear them, with 14-byte threshold
    __outbyte( 0x03F8 + 4, 0x0B );    // IRQs enabled, RTS/DSR set
    
    __outbyte( 0x03F8 + 4, 0x1E );    // Set in loopback mode, test the serial chip
    __outbyte( 0x03F8 + 0, 0xAE );    // Test serial chip (send byte 0xAE and check if serial returns same byte)
    
    byte = __inbyte( 0x03F8 ); // Confirmed: return result is 0xAE
    
    __outbyte( 0x03F8 + 4, 0x0F );

PuTTY is currently being used in a baseline attempt to record output from COM1, though the same result--which is no data--can be seen in redirecting the port from within VMware. The settings within PuTTY are as follows:

• Serial mode

• Serial line: COM1

• Speed (baud): 115200

• Data bits: 8

• Stop bits: 1

• Parity: None

• Flow control: XON/XOFF (Default)

Different flow control values were used in testing, which included (from the option set): None, XON/XOFF, RTS/CTS, DSR/DTR. I attempted to play with these values via the modem control register, but didn't have any luck after setting the fields and playing with the flow control configuration.

The physical processor backing the VM is an Intel i9-13900K, running on VMware Workstation 17 Pro (17.5.0 build-22583795).

Thank you for your time. I really appreciate it, and all the best.

[Octocontrabass]

Code: Select all

    rep outsb dx, byte ptr [rsi]

Virtual machines probably won't care, but this may be too fast for a physical serial port. Also, which ABI are you using?

I can't seem to observe any sort of serial port output produced given the configuration above.

Have you tried a different virtual machine? QEMU and Bochs are both excellent for debugging.

[samseaborn]

Code: Select all

    rep outsb dx, byte ptr [rsi]

Virtual machines probably won't care, but this may be too fast for a physical serial port. Also, which ABI are you using?

Considering my microkernel is currently initialized and executed by a Windows 11 guest, I believe I am using the Microsoft x64 ABI; as is detailed in the links below.
https://learn.microsoft.com/en-us/cpp/b ... w=msvc-170
https://learn.microsoft.com/en-us/cpp/b ... w=msvc-160

I am very curious what you mean by this form of output being too fast for physical hardware. Do you believe lowering the buad rate could help? Is there something you've seen or experienced which would lead you to change how the code posted outputs to COM? I do intend on running this microkernel on physical hardware once things are more stable in the virtualized environments, so the idea that this may only work in a VM is very interesting to me.

I can't seem to observe any sort of serial port output produced given the configuration above.

Have you tried a different virtual machine? QEMU and Bochs are both excellent for debugging.

I would love to make use of Bochs, though my microkernel project requires a host OS to initialize and execute it (similar to something you'd see in a late-launch hypervisor). In my experience, Bochs was always far too slow to run a Windows 10+ guest, though this very easily could have been an issue on my end with configuring the guest. VirtualBox doesn't work with my company's custom tooling for debugging guest operating systems because of some issues with the TPM when the guest is >= Windows 11, so I haven't given that solution a try yet either. Given these limitations with other emulators and virtual machines that were specifically designed for Windows, I haven't given QEMU a fair shot as I assumed it was generally out of the question.

I will try to make some progress on the Bochs front though, and see if there is anything that can be gathered from the modem control or line status registers. I have an inkling suspicion this could be an issue related to the PuTTY configuration.

Regardless, thank you for your reply and for the help I appreciate you!

[samseaborn]

I have an update to the situation which may or may not be the fault of VMware.

I have created a successful logging condition built around another version of PuTTY running on the guest machine. This additional version of PuTTY is listening on the COM1 port with the standard settings above (COM1, 115200 baud rate, 8-bit data size, 1-stop bit, no parity, no flow control). The hardware exposed to the guest now redirects to a named pipe which is consumed by another PuTTY instance on the host with the same settings minus the target name of the COM1 port (which is instead the name of the pipe setup by VMware).

This setup successfully outputs information sent to the serial port by the guest OS on the host, though if this nested instance of PuTTY is not included in the setup chain--which outputs nothing regardless of the surrounding configuration (i.e. no COM1 output can ever be observed on the guest OS)--the logging on the host will cease.

I'm uncertain if this is an issue with my configuration of the serial port (such as some buffering or synchronization bug that is only then resolved with another consumer sitting on the port), or if this is a VMware issue, or if perhaps there is another reason why this behavior may be observed in the way it is. Perhaps this is indicative of some obvious issue with the serial port configuration?

I've additionally reached out to individuals at Broadcom with questions about how VMware handles this case and will post back if any of them confirm the issue.

[Octocontrabass]

Considering my microkernel is currently initialized and executed by a Windows 11 guest,

What? Why? That seems like it'd be way more trouble than it's worth.

I am very curious what you mean by this form of output being too fast for physical hardware.

I mean the UART has a finite buffer to store data before physically transmitting it across the wire, and REP OUTSB doesn't wait for room in that buffer before trying to fill it anyway. You need to either poll the line status register or set up and wait for an interrupt to make sure none of the data you're writing gets lost.

Do you believe lowering the buad rate could help?

It'll make the problem worse, since it'll take longer for data to leave the transmit buffer.

I haven't given QEMU a fair shot as I assumed it was generally out of the question.

QEMU should be able to run Windows 11 better than Bochs.

[samseaborn]

I mean the UART has a finite buffer to store data before physically transmitting it across the wire, and REP OUTSB doesn't wait for room in that buffer before trying to fill it anyway. You need to either poll the line status register or set up and wait for an interrupt to make sure none of the data you're writing gets lost.

Thank you for the insight. I think I may end up fleshing out a more robust output byte function and then base my output byte string function on that. Perhaps I can do some sort of polling on the line status register within this output byte function like you mentioned.

QEMU should be able to run Windows 11 better than Bochs.

Yea I can certainly find the time to experiment with both of them at the moment. However, given the successful test case above with VMware, I think it might make more sense in my project--at least at the moment--to hammer out whatever outstanding issues are the way of making this serial communication more seamless.

Do you have any thoughts on the behavior of the test case I provided above? Does this seem to you like something that would be reflective of a bad serial port configuration?

Thanks again for all the input!

[Octocontrabass]

Do you have any thoughts on the behavior of the test case I provided above?

Sounds like the Windows driver for that serial port interferes with your kernel trying to use that serial port. Which brings me back to an earlier question: why are you using Windows to load your kernel instead of a normal bootloader?

[samseaborn]

Do you have any thoughts on the behavior of the test case I provided above?

Sounds like the Windows driver for that serial port interferes with your kernel trying to use that serial port. Which brings me back to an earlier question: why are you using Windows to load your kernel instead of a normal bootloader?

Well, like I mentioned earlier, the project is structured in the same fashion a late-launch hypervisor would be. I apologize if that wasn't as clear before as I intended for it to be. This is referring to the type of hypervisors you might see antivirus services using to create a minimalistic operating environment powered by processor virtualization technology that is isolated from the guest. Such services are a completely valid use case for this type of microkernel structure, and while my project differs from these antivirus services in its intention (which is closer to a certain Citrix application), the implementation is certainly similar. I hope that helps to clear up any confusion regarding why the this project is structured the way it is. Having the microkernel environment run parallel to an existing host is entirely intentional.

Onto an update regarding the aforementioned research on the Line Status and Modem Status registers: when the PuTTY instance is present on the guest and configured to consume from the COM1 serial port, the Line Status Register is read as 0x60 (Transmitter Holding Register Empty (THRE) bit set + Transmitter Empty (TEMPT) bit set), and the Modem Status Register is read as 0xB0 (Clear to Send (CTS) bit set + Data Set Ready (DSR) bit set + Data Carrier Detected (DCD) bit set). Finally, reading from the COM1 port accurately returns 0x00 when no data has been transmitted.

When the auxiliary PuTTY instance is not present on the guest actively consuming from COM1, the Line Status Register is read as 0xFF, and the Modem Status Register is read as 0xFF. Additionally, reading from the COM1 port yields another 0xFF.

I'm yet to find any information online regarding the 0xFF return result on these registers, which is making me believe this may be something weird to do with VMware. The conflicting status information reflected with all these bits set also gives credence to that theory (i.e. the Line Status Register having both the Transmitter Holding Register Empty (THRE) and Data Ready (DR) bits set concurrently).

My thought was maybe the auxiliary PuTTY instance was configuring the serial port in a way my kernel code wasn't, and that's why the data managed to get through, but after running a test where the port is setup by my kernel code after the PuTTY instance would have setup the port, the data still passes through to the host just fine.

I have already played with the values in the Modem Control Register. Mostly the Data Terminal Ready (DTR), Request to Send (RTS), and Out 2 bits. This, coupled with correct port configurations on the both Windows 11 instances, did not lead to any changes in the situation.

At this point I'm not sure what PuTTY could be doing differently when interacting with the serial port that allows data to get from the guest to the host; or why that instance of PuTTY is never able to see the COM output itself. Even when configuring the VM to pass through the COM1 port directly (rather that redirecting it through a named pipe) results in a 0x60 from the Line Status Register, a 0x00 from the Modem Status Register, and no output from within the PuTTY instance on the guest.

Edit: it appears the values I output to the COM registers are not being written to said registers. Executing an OUT to the Line Control Register, for instance, does not return the written byte when executing an IN on the same Line Control Register.

[Octocontrabass]

When the auxiliary PuTTY instance is not present on the guest actively consuming from COM1, the Line Status Register is read as 0xFF, and the Modem Status Register is read as 0xFF. Additionally, reading from the COM1 port yields another 0xFF.

That sounds like open bus. The Windows driver must be disabling or remapping the serial port when it thinks nothing is using it.

You might have to go through the normal Windows APIs for accessing the serial port if you don't want to boot your microkernel before you boot Windows.

[samseaborn]

When the auxiliary PuTTY instance is not present on the guest actively consuming from COM1, the Line Status Register is read as 0xFF, and the Modem Status Register is read as 0xFF. Additionally, reading from the COM1 port yields another 0xFF.

That sounds like open bus. The Windows driver must be disabling or remapping the serial port when it thinks nothing is using it.

You might have to go through the normal Windows APIs for accessing the serial port if you don't want to boot your microkernel before you boot Windows.

I know it doesn't seem at all like we're going to get anywhere with this, but just on the off-chance that maybe something cool can happen here (and if we can hopefully aid in somebody else's attempt at solving a similar issue): do you think there's any reasonable way to enable COM1 serial port communication--as would be desirable with the configuration above--without using Windows-specific APIs?

You may not be interested, but for those who search and find this thread in the future: here is about where I left off. The Windows serial driver is not open source, but you can effectively view pretty much all of the code for it given that the public debugging symbols have nearly 100% code coverage and the WDK, WRK, and XP source code all provide what seem to be matching function definitions for pieces of Serial.sys's source code. I can also confirm many of these source snippets are still accurate in recent Windows 11 releases.

Code: Select all

DriverEntry:   fffff8017d268010serial!GsDriverEntry
DriverStartIo: 00000000
DriverUnload:  fffff8017d25de60serial!SerialUnload
AddDevice:     fffff8017d25e710serial!SerialAddDevice

Dispatch routines:
[00] IRP_MJ_CREATE                      fffff8017d25ded0serial!SerialCreateOpen
[01] IRP_MJ_CREATE_NAMED_PIPE           fffff801787516b0nt!IopInvalidDeviceRequest
[02] IRP_MJ_CLOSE                       fffff8017d264410serial!SerialClose
[03] IRP_MJ_READ                        fffff8017d264e60serial!SerialRead
[04] IRP_MJ_WRITE                       fffff8017d266980serial!SerialWrite
[05] IRP_MJ_QUERY_INFORMATION           fffff8017d25e2d0serial!SerialQueryInformationFile
[06] IRP_MJ_SET_INFORMATION             fffff8017d25e420serial!SerialSetInformationFile
[07] IRP_MJ_QUERY_EA                    fffff801787516b0nt!IopInvalidDeviceRequest
[08] IRP_MJ_SET_EA                      fffff801787516b0nt!IopInvalidDeviceRequest
[09] IRP_MJ_FLUSH_BUFFERS               fffff8017d25d010serial!SerialFlush
[0a] IRP_MJ_QUERY_VOLUME_INFORMATION    fffff801787516b0nt!IopInvalidDeviceRequest
[0b] IRP_MJ_SET_VOLUME_INFORMATION      fffff801787516b0nt!IopInvalidDeviceRequest
[0c] IRP_MJ_DIRECTORY_CONTROL           fffff801787516b0nt!IopInvalidDeviceRequest
[0d] IRP_MJ_FILE_SYSTEM_CONTROL         fffff801787516b0nt!IopInvalidDeviceRequest
[0e] IRP_MJ_DEVICE_CONTROL              fffff8017d262c00serial!SerialIoControl
[0f] IRP_MJ_INTERNAL_DEVICE_CONTROL     fffff8017d2628d0serial!SerialInternalIoControl
[10] IRP_MJ_SHUTDOWN                    fffff801787516b0nt!IopInvalidDeviceRequest
[11] IRP_MJ_LOCK_CONTROL                fffff801787516b0nt!IopInvalidDeviceRequest
[12] IRP_MJ_CLEANUP                     fffff8017d264320serial!SerialCleanup
[13] IRP_MJ_CREATE_MAILSLOT             fffff801787516b0nt!IopInvalidDeviceRequest
[14] IRP_MJ_QUERY_SECURITY              fffff801787516b0nt!IopInvalidDeviceRequest
[15] IRP_MJ_SET_SECURITY                fffff801787516b0nt!IopInvalidDeviceRequest
[16] IRP_MJ_POWER                       fffff8017d261690serial!SerialPowerDispatch
[17] IRP_MJ_SYSTEM_CONTROL              fffff8017d261550serial!SerialSystemControlDispatch
[18] IRP_MJ_DEVICE_CHANGE               fffff801787516b0nt!IopInvalidDeviceRequest
[19] IRP_MJ_QUERY_QUOTA                 fffff801787516b0nt!IopInvalidDeviceRequest
[1a] IRP_MJ_SET_QUOTA                   fffff801787516b0nt!IopInvalidDeviceRequest
[1b] IRP_MJ_PNP                         fffff8017d2602a0serial!SerialPnpDispatch

All of the port configuration commands seem to go through that IOCTL endpoint (SerialSystemControlDispatch), which has a different name in the source code linked below but recent symbols match it 1:1.

Serial.sys!SerialCreateOpen - https://github.com/tongzx/nt5src/blob/d ... clos.c#L89

Serial.sys!SerialClose - https://github.com/tongzx/nt5src/blob/d ... los.c#L404

SerialClose device tear down code - https://github.com/tongzx/nt5src/blob/d ... los.c#L519

Serial.sys!SerialDisableUART - https://github.com/tongzx/nt5src/blob/d ... los.c#L934

https://github.com/microsoft/Windows-dr ... ial/serial
https://learn.microsoft.com/en-us/windo ... nd-serenum

Regardless, I appreciate the help in formulating my path forward Thanks!

[feryno]

I'm hypervisor developer too. I came to the same problem some time ago and was unable to solve it either. The hypervisor could be started from running OS using a driver, or prior OS (from UEFI or BIOS) and only then load an OS as a guest. All my devel machines have physical serial ports on motherboards (no additional PCI cards with serial ports needed). When starting the hypervisor from BIOS or UEFI the serial ports are working OK (like sending initial hello message) but only temporarily, once the guest OS is started, the serial ports no more working (no transmission of data possible either the initializing sequence similar to your one does nothing). Seems the guest OS may reprogram the serial port device or set it into some strange mode which I do not know (couldn't it be like DMA mode or remapping into memory mapped device writes instead of IO port writes? - just my guesses, I did not investigate it further). When the hypervisor is started from running OS (using a driver) I can't call OS drivers stack / use OS serial port driver because the hypervisor is completely isolated, have own paging tables separated from OS paging tables, private GDT, IDT, GS base and so one... the physical memory where it lives is hidden from guest OS (using EPT on Intel / RVI = nested paging on AMD). For UEFI / BIOS started hypervisor I even tried to solve the problem by preventing guest OS to write into serial io ports (using vm exit IO map to prevent writes to the io ports) but that did not help either, once the guest OS was started it very likely reprograms the serial ports in an unknown way so io ports were not working for my hypervisor. The same behavior for Intel as well AMD, the same on old hardware (more than 10 years old) as well new one. So for some necessary situations I just toggle PS/2 keyboard LEDs to know that something happened or send data to IO port 0x80 or use legacy speaker device via io ports 0x42, 0x43, 0x61 to play some melody like in >30 years old ms dos games (Doom II still had this sound output mode). But speakers support was removed on recent hardware too, although motherboards still have speaker connector pins and I soldered few speakers / buzzers manually as they are not sold anymore either. PS/2 keyboard ports and PS/2 keyboard LEDs are also rare today but still available (server motherboards due to compatibility or gamer motherboards as PS/2 does not have USB latency). I have some limited success via io port 0x80 on ASUS and ASROCK motherboards to transfer some data, ms win does not touch this port, Linux usually writes there a zero value only once during OS init, UEFI runtime could send data via this port too and interfere.
If your hypervisor is separated from ms win OS you could also develop some alternative way of sending data, e.g. writing into some separated physical memory pages, toggle PS/2 keyboard LEDs, play something through a speaker device, use io port 0x80 on hardware which supports it. Seems like stoneage methods but I did not find anything better yet. When running hypervisor under vmware (vmware parent, my hypervisor child) I used to write some info into physical memory page at 0-0xFFF (unused on UEFI systems or holding 16 bit real mode IDT on BIOS systems which is not used when OS is already running), suspend VM execution and inspect the VM RAM saved to the disk as a file. Having working serial port would be the best solution but I was unable to do that - I observed the same as you already wrote - it appears the values I output to the COM registers are not being written to said registers.

[Octocontrabass]

do you think there's any reasonable way to enable COM1 serial port communication--as would be desirable with the configuration above--without using Windows-specific APIs?

I would guess Windows is using ACPI to turn off the serial port when nothing is using it. I don't think there's any reasonable way to get around that, but there are plenty of unreasonable ways, like intercepting the ACPI tables before Windows boots and rewriting them to better fit your hypervisor's needs, or writing lots of chipset drivers so you can intercept and override Windows' attempts at power management.

Actually, wait, there is one reasonable way: write a virtual machine with virtual hardware for Windows.

[feryno]

Octocontrabass - that's very likely the case of not working serial port. The ACPI needs to be intercepted also because of S3 sleep /resume - the CPU is woken up with disabled virtualization which effectively kills any hypervisor (Intel as well AMD). If the hypervisor is a ms windows driver you can use \Callback\PowerState to resurrect.