Guide on how to test network of OS with Qemu ?

[wichtounet]

Hi,

My next goal for my OS is to have networking working. I have been able to add a network interface to Qemu:

-net nic,vlan=0,model=rtl8139

and from this, I have been able to detect the PCI Configuration and obtain the MAC Address. Now, I want to see if packets are getting received, but I don't see how a) to configure Qemu for this and b) to send packets to the guest debug it.

I know that there are a lot of posts on this subjects and I've a tried a lot of commands for Qemu for bridge or tap, but none of them worked. I ended up with operation not permitted errors, obscure errors (256?) or simply not having network on the host anymore. Moreover, all the posts that I have imply being on Ubuntu, which I'm not (Gentoo), so I need standard commands rather Ubuntu network configuration.

Does someone has a comprehensive guide on how to do this ? Or a good link ?

Thanks

[Kevin]

Well, you definitely need some network backend, otherwise your packets go to nowhere. The most powerful version is indeed using a tap device, but it's notoriously tricky to set up as you already noticed. In theory, it's easy: Set up a tap device that is owned by your user (with tunctl or ip), optionally include it in a bridge with your physical network card (brctl) and then give it to qemu.

Other options include "-net user", which emulates a virtual router that NATs you to the host network (by default, the router is 10.0.2.2). The disadvantage here is that you can't simply attach Wireshark to the device. However, with "-net dump" you can still capture the traffic in your virtual network. Maybe this is good enough for you.

[iansjack]

Use the -net dump option to dump packets to a file.

[wichtounet]

Thanks guys.

Well, you definitely need some network backend, otherwise your packets go to nowhere. The most powerful version is indeed using a tap device, but it's notoriously tricky to set up as you already noticed. In theory, it's easy: Set up a tap device that is owned by your user (with tunctl or ip), optionally include it in a bridge with your physical network card (brctl) and then give it to qemu.

I've finally been able to configure it

Here is the final qemu command I'm using:

Code: Select all

sudo qemu-system-x86_64 -enable-kvm -cpu host -serial file:virtual.log -netdev tap,helper=/usr/libexec/qemu-bridge-helper,id=thor_net0 -device rtl8139,netdev=thor_net0,id=thor_nic0 -vga std -hda hdd.img &

The network relevant part being:

Code: Select all

-netdev tap,helper=/usr/libexec/qemu-bridge-helper,id=thor_net0 -device rtl8139,netdev=thor_net0,id=thor_nic0

I've discovered the bridge helper from Qemu: http://wiki.qemu.org/Features/HelperNetworking It does almost everything
The only thing necessary is to create the bridge:

Code: Select all

sudo brctl addbr br0

To send a packet, I'm using

Code: Select all

ping -I tap0 2.2.2.2

Now I have to find a tool to generate some packets for me on a specific interface.

Other options include "-net user", which emulates a virtual router that NATs you to the host network (by default, the router is 10.0.2.2). The disadvantage here is that you can't simply attach Wireshark to the device. However, with "-net dump" you can still capture the traffic in your virtual network. Maybe this is good enough for you.

I've checked documentation on "user networking" mode, but the problem is that it only supports TCP and UDP and I wanted to start with ICMP, but I'll maybe start to investigate this further.

Use the -net dump option to dump packets to a file.

My problem is more to send a packet to the VM right now, but thanks

[Kevin]

Now I have to find a tool to generate some packets for me on a specific interface.

Well, with a correctly working bridge, all packets are by definition visible on all interfaces that are connected in this bridge.

Edit: Oh, you may not be including your physical network card in the bridge, I don't think qemu-bridge-helper does that. So you can either do that and effectively make your VM part of the physical network (brctl add; and then I seem to remember that you need to clear the IP address on your ethernet interface and get one for the bridge interface instead); or you leave your physical network alone, assign the bridge an IP address and subnet so that this specific subnet is routed to the bridge (with only a single tap device in it), and everything else still goes to your physical interface. If you later want to access the internet from your VM, you could do normal NAT.

[Boris]

Hi,
If you want to generate packets, try looking at the man page for packet(7)

You will want to use AF_PACKET to test raw Ethernet stuff.
Once you have arp working, use SOCK_RAW to generate raw IP packets.

[wichtounet]

It seems I'm still not there at all finally

If I ping on tap0, the only thing I'm receiving is ARP request. Even if I reply, I dont' receive any ICMP packets
I also always receive some IPV6 packets at random intervals for some reason.

Now I have to find a tool to generate some packets for me on a specific interface.

Well, with a correctly working bridge, all packets are by definition visible on all interfaces that are connected in this bridge.

Edit: Oh, you may not be including your physical network card in the bridge, I don't think qemu-bridge-helper does that. So you can either do that and effectively make your VM part of the physical network (brctl add; and then I seem to remember that you need to clear the IP address on your ethernet interface and get one for the bridge interface instead); or you leave your physical network alone, assign the bridge an IP address and subnet so that this specific subnet is routed to the bridge (with only a single tap device in it), and everything else still goes to your physical interface. If you later want to access the internet from your VM, you could do normal NAT.

No, I did not add my physical card to the bridge. When I do, it does not seem to pass all packets anyway and when I do, I don't have internet anymore on the host computer :s
If I assign an IP to the bridge or to tap0 and put it up, ping indicates that the interface is up, but no packet is transferred to the guest...

Another issue that I seem to have is that the MAC address I read on the interface card is neither that of the tap0 neither of the bridge. Shouldn't that be the same mac as the tap0 interface ?

It seems to me that network testing an OS is harded than to code the network stack itself

Hi,
If you want to generate packets, try looking at the man page for packet(7)

You will want to use AF_PACKET to test raw Ethernet stuff.
Once you have arp working, use SOCK_RAW to generate raw IP packets.

Thanks, I'll take a look

[Kevin]

If I ping on tap0, the only thing I'm receiving is ARP request. Even if I reply, I dont' receive any ICMP packets

Did you check (e.g. with Wireshark) that your ARP reply is indeed correct?

No, I did not add my physical card to the bridge. When I do, it does not seem to pass all packets anyway and when I do, I don't have internet anymore on the host computer :s

That's obviously not what should happen. Did you remove the IP address from your physical interface as I said? When I configured things incorrectly, I occasionally lost my network connection, too, but once configured correctly, it definitely works. And I seem to remember that "correct configuration" means that the physical interface and the tap interface don't have an IP assigned, but the bridge has.

Another issue that I seem to have is that the MAC address I read on the interface card is neither that of the tap0 neither of the bridge. Shouldn't that be the same mac as the tap0 interface ?

No, the tap0 interface is a virtual network device of your host. If you don't bridge it to the physical network, this MAC address is what your VM has to use to send something to the host. The guest MAC address is different and can be configured as an option to -net in qemu.

[wichtounet]

Thanks Kevin!

If I ping on tap0, the only thing I'm receiving is ARP request. Even if I reply, I dont' receive any ICMP packets

Did you check (e.g. with Wireshark) that your ARP reply is indeed correct?

It seems fine in Wireshark:

Code: Select all

1	0.000000000	fe:48:e4:24:85:ce	Broadcast	ARP	42	Who has 64.65.66.67? Tell 192.168.20.201
2	0.045463542	RealtekU_12:34:56	fe:48:e4:24:85:ce	ARP	42	64.65.66.67 is at 52:54:00:12:34:56

But it is done three times, so there must be something wrong somewhere.

No, I did not add my physical card to the bridge. When I do, it does not seem to pass all packets anyway and when I do, I don't have internet anymore on the host computer :s

That's obviously not what should happen. Did you remove the IP address from your physical interface as I said? When I configured things incorrectly, I occasionally lost my network connection, too, but once configured correctly, it definitely works. And I seem to remember that "correct configuration" means that the physical interface and the tap interface don't have an IP assigned, but the bridge has.

I tried, but in my case, I did not have have any internet when I did that. There must be a configuration error somewhere, I'll check again later.

Another issue that I seem to have is that the MAC address I read on the interface card is neither that of the tap0 neither of the bridge. Shouldn't that be the same mac as the tap0 interface ?

No, the tap0 interface is a virtual network device of your host. If you don't bridge it to the physical network, this MAC address is what your VM has to use to send something to the host. The guest MAC address is different and can be configured as an option to -net in qemu.

Ok, that works, at least

[wichtounet]

I'm still working on network support and I now have a good support (IP,TCP/DNS,HTTP,DHCP...).

There is still one thing I haven't been able to figure out is how to have several machines in the same 'user' network. If I create the first virtual machine with a "socket listen" and then the second virtual machine with "socket connect", I have a lot of weird TCP packets in Wireshark and I cannot make sense of them (maybe they come from me but I don't see how).

Does someone has a good solution for this ?

[mariuszp]

if you can show a screenshot of wireshark or give the CAP file we might be able to help.

[wichtounet]

if you can show a screenshot of wireshark or give the CAP file we might be able to help.

Sure, I'll do that once I'm back at home.

[wichtounet]

Here is the pcap file: https://baptiste-wicht.ch/thor.pcap

The RST packets are not from me and the first FIN,ACK packets neither. Pretty painful when I'm debugging TCP.

[mariuszp]

Here is the pcap file: https://baptiste-wicht.ch/thor.pcap

The RST packets are not from me and the first FIN,ACK packets neither. Pretty painful when I'm debugging TCP.

1) are you sure one of those addresses isn't QEMU's DHCP server?
2) is some form of NAT involved?
3) is there a firewall involved?

[mariuszp]

In fact, is your TCP segment structure correct? Perhaps you are accidentally setting the wrong bit.