[Solved] Passing network traffic from macOS to QEMU

[codyd51]

Hello!

Apologies for any errors in terminology or missing relevant info, as I'm quite new to networking in general.

I'm writing an RTL8139 driver in my OS, and am having a truly hard time getting the TAP device working correctly on macOS on WiFi with QEMU v5.2.0. In brief: I can transmit packets and they are visible on the "tap0" interface, but do not receive any packets.

I've set up a bridge with my main network interface, en0, and the TAP device, tap0. "bridge0" is used for Thunderbolt connectivity, so I've created "bridge2" for my purposes:

QEMU config:

Code: Select all

-nic tap,model=rtl8139,script=./qemu-ifup.sh,downscript=./qemu-ifdown.sh,id=u1 -object filter-dump,id=f1,netdev=u1,file=dump.dat

qemu-ifup.sh:

Code: Select all

#!/bin/bash
# https://gist.github.com/EmbeddedAndroid/6572715
# https://gist.github.com/artembeloglazov/db8c16efc91443955fca
sysctl -w net.link.ether.inet.proxyall=1
sysctl -w net.inet.ip.forwarding=1
sysctl -w net.inet.ip.fw.enable=1
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.link.tap.user_open=1
sysctl -w net.link.tap.up__on__open=1
ifconfig bridge2 create
ifconfig bridge2 inet 192.168.13.1/24
ifconfig bridge2 addm en0 addm $1
ifconfig bridge2 up
ifconfig en0 up
ifconfig $1 up

I am in fact able to transmit packets within my OS, and these packets are visible in Wireshark when snooping the tap0 interface (though the packet only contains 0x41 repeated at the moment!). The issue is that QEMU is not receiving any packets, even though a wide breadth of network traffic appears in Wireshark or tcpdump. I have recompiled QEMU with the debug flag for this driver enabled, and I don't get any extra output except while initializing the driver or when I transmit a packet.

I've scoured a lot of info online about setting this up and have yet to resolve it. I saw someone imply that the WiFi card in MacBooks simply doesn't support this, so wanted to check-in here to see if it is, in fact, possible.

Does anyone have any relevant experience they can throw in? Thanks very much!

[xeyes]

The tap setup with qemu is a bit tricky, but it is easy to verify by answering this simple question:

Does your tap setup work if you run a "known good guest"?

like, run some liveCD of Linux and try to ssh into it from the host through your tap.

[codyd51]

The tap setup with qemu is a bit tricky, but it is easy to verify by answering this simple question:

Does your tap setup work if you run a "known good guest"?

Good idea! Thank you. I tried it and was unable to get it to work, but this test exists at the intersection of two things I'm unfamiliar with (network routing and Linux), so I don't take that as conclusive!

The plot thickens (from https://support.apple.com/en-au/guide/m ... h43557/mac):

Note: You can’t use bridging if the physical network device used for bridging is a wireless device.

I also see that the TAP device is assigned a random MAC when it's opened by QEMU (https://sourceforge.net/p/tuntaposx/cod ... ap.cc#l150), and that the emulated NIC created by QEMU has a different MAC. Just as a test, I'm now re-assigning the MAC of the TAP in the qemu-up script to match the QEMU NIC.

For now, I've purchased a USB WiFi adapter that arrives tomorrow, and will try doing all this with an ethernet cable plugged between my router and Macbook.

[codyd51]

No dice when using an ethernet cable either.

[xeyes]

The tap setup with qemu is a bit tricky, but it is easy to verify by answering this simple question:

Does your tap setup work if you run a "known good guest"?

Good idea! Thank you. I tried it and was unable to get it to work, but this test exists at the intersection of two things I'm unfamiliar with (network routing and Linux), so I don't take that as conclusive!

The plot thickens (from https://support.apple.com/en-au/guide/m ... h43557/mac):

No worries, I'm also not familiar with this but I can recommend this guide to you, which is the one that enabled me to connect to a guest (Linux, SSH or other servers).

While it is not exactly what you want (I don't think it bridges the real interface to the guest, just between host and guest), it might still be of some help to you.

https://alberand.com/host-only-networki ... visor.html

Good luck!

[codyd51]

I’ve simplified the failing case some more. If I write my own application to consume packets from the tap, I am able to successfully receive packets:

Code: Select all

# In `sudo python3`
f = open("/dev/tap0", "r")
while True:
    r = select([f.fileno()], [], [])[0][0]
    packet = os.read(f.fileno(), 4000)
    print(packet)

# In `/bin/bash`
sudo ifconfig tap1 10.12.0.2 10.12.0.1
ping 10.12.0.1

# The Python program prints packets from the tap

However, if I spawn QEMU instead of my python3 script and run the same bash commands, QEMU doesn’t output anything about having received a packet. I am being led further to think that there is some QEMU issue receiving packets on a tap with macOS, of which I’m running Big Sur v11.0.1 and 11.1. I am going to dig into the QEMU source a bit and add some logs where possible.

[bzt]

You ifdown script is using "sysctl" and "ifconfig". Both are Linux specific commands, and since neither your host nor your guest is Linux, I doubt they will work correctly.
(MacOS might actually have an incompatible "ifconfig" since it's a BSD (probably with different switches and flags), but I'm 100% sure it doesn't have the same "sysctl" flags as Linux. The man page does not list "net.link.tap.user_open" for example) So are you sure those are the correct flags?

Cheers,
bzt

[codyd51]

You ifdown script is using "sysctl" and "ifconfig". Both are Linux specific commands, and since neither your host nor your guest is Linux, I doubt they will work correctly.
(MacOS might actually have an incompatible "ifconfig" since it's a BSD (probably with different switches and flags), but I'm 100% sure it doesn't have the same "sysctl" flags as Linux. The man page does not list "net.link.tap.user_open" for example) So are you sure those are the correct flags?

Indeed, macOS ships a clearly-different ifconfig from the one referenced by lots of discourse online: mine has different switches and different output from the (plural?) Linux one.

Similarly, as you correctly guessed, the available sysctl's are different. Some of the ones in my post, cobbled together from info online, are invalid on macOS, but sysctl simply complains about it and no-op's when an invalid name is provided. I fiddled with a few more switches to no avail after inspecting the output of "sysctl -a | grep -i net".

Currently, I'm knee-deep in qemu. I've learned that it uses glib for its event loop, and has an internal system called "aio" used to dispatch events in respond to glib wakeups. The tap's file descriptor is certainly provided to glib to poll on, but QEMU does not seem to react when new data appears on the tap. I'm still digging...

Thanks for your input

[codyd51]

As another "quick test", I wrote an equivalent C program to the Python script above that opens the tap and select()'s on it. When I configure the tap via ifconfig, data appears and select() unblocks. No reaction within QEMU, but I've yet to dig deep enough into it to isolate where the wakeup should happen -- there are lots of internal systems even just in the small bit I'm touching! (notify, timers, bottomhalf, aio, iohandler... it's enough to drive anyone to sticky notes)

[codyd51]

I've finally gotten to the bottom of this, after hacking around in QEMU for a while and speaking with a couple of very helpful QEMU contributors in their IRC.

As mentioned above, select() works fine on the http://tuntaposx.sourceforge.net driver tap device. However, QEMU (via glib) doesn't use select() to wait on file-descriptors: it uses poll(). And, as it turns out, macOS does not support poll() on devices https://developer.apple.com/library/arc ... oll.2.html:

BUGS
The poll() system call currently does not support devices.

A couple of years ago, macOS introduced a framework intended to provide support for virtualization software to pass packets to and from the host: vmnet.framework https://developer.apple.com/documentati ... guage=objc.

I am now working on a QEMU patch implementing a new "netdev" backed by the native vmnet.framework. Receiving packets works well, and I'm currently adding support for fun options (like providing a MAC address instead of letting the kernel allocate one).

A bit disappointing, but I'm glad to finally understand what was going on here and what the path forward is. I've marked this post as [Solved].

[BarfSchwartz]

You would’ve needed to route from the bridge to your real network interface. Pf would help with that. Don’t put the wireless interface into the bridge.

[codyd51]

Update: My QEMU patch to implement a macOS/vmnet network backend has been in review for a while, and I've been using it to continue building my network stack: https://lists.gnu.org/archive/html/qemu ... 04637.html

So far, I've got ARP, IP, UDP and DNS, and am about to break ground on TCP! It's so gratifying to be discovering devices on my local network.

If all goes well, this feature will be included in QEMU 6.0. I hope this patch helps someone else on macOS who was in a similar situation to me!

[bzt]

Update: My QEMU patch to implement a macOS/vmnet network backend has been in review for a while

Well done! That's the power of Open Source! Don't be alarmed, I'm a qemu contributor myself, I had some patches for which I had to wait months... On the other hand, others (like my ARM QA7 timer patch) was reviewed within a few weeks. No worries, they will get there eventually!

Again, well done!

Cheers,
bzt

[Princekin]

that help me！thanks a lot！

[Princekin]

Code: Select all

If all goes well, this feature will be included in QEMU 6.0. I hope this patch helps someone else on macOS who was in a similar situation to me!

my qemu-system-i386 is

Code: Select all

QEMU emulator version 6.2.0

, still has this problem, it seems the qemu does not fix this bug in 6.x version qemu!