Multiple Network Adapters

[SpyderTL]

I'm feeling a bit too lazy at the moment to research this properly, so I thought that someone here could give me a quick answer.

I'm trying to decide whether a TCP/IP active connection should be bound to a network adapter or not. I'm assuming windows does bind connections to adapters, since disabling one adapter will kill any connections on that adapter, rather than elegantly moving them to another adapter, or just attempting to use whatever adapter is available.

So, it it even feasable to put all active connections in a bucket, and every packet that is sent will just use any adapter that is on the appropriate network. Would you need to remember the "original" ethernet and/or ip address, and use them when sending packets from the new card? I assume so, since the remote machine is expecting packets from a specific ip address/port.

Any thoughts on this?

[Brendan]

Hi,

So, it it even feasable to put all active connections in a bucket, and every packet that is sent will just use any adapter that is on the appropriate network. Would you need to remember the "original" ethernet and/or ip address, and use them when sending packets from the new card? I assume so, since the remote machine is expecting packets from a specific ip address/port.

The words you're looking for are "ethernet bonding".

The basic idea is to make 2 or more physical ethernet devices appear as one logical ethernet device for some purpose (e.g. improved performance or fault tolerance or both). It's the same basic idea as RAID (making 2 or more physical disks appear as one logical disk). This sounds simple, but there's different ways it can be done and various complications for each of those different ways.

The best place to start might be Linux Ethernet Bonding Driver HOWTO, as this will give you a reasonable idea of the capabilities and problems.

I'd also assume that for the purpose of improving performance it's probably better/easier to do load balancing at a higher level (e.g. in the IP layer) and let routing protocols figure out which ethernet card to use. Ethernet bonding happens at a much lower level (raw ethernet packets).


Cheers,

Brendan

[SpyderTL]

Hey Brendan,

That's somewhat close to what I've been thinking about. However, I am thinking more about the OS pooling all network devices, instead of the user specifically configuring individual device behavior. However, the link you provided touches one one of the problems that I am concerned about...

8.1 Adventures in Routing
-------------------------

When bonding is configured, it is important that the slave
devices not have routes that supersede routes of the master (or,
generally, not have routes at all). For example, suppose the bonding
device bond0 has two slaves, eth0 and eth1, and the routing table is
as follows:

Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth0
10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth1
10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 bond0
127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo

This routing configuration will likely still update the
receive/transmit times in the driver (needed by the ARP monitor), but
may bypass the bonding driver (because outgoing traffic to, in this
case, another host on network 10 would use eth0 or eth1 before bond0).

The ARP monitor (and ARP itself) may become confused by this
configuration, because ARP requests (generated by the ARP monitor)
will be sent on one interface (bond0), but the corresponding reply
will arrive on a different interface (eth0). This reply looks to ARP
as an unsolicited ARP reply (because ARP matches replies on an
interface basis), and is discarded. The MII monitor is not affected
by the state of the routing table.

The solution here is simply to insure that slaves do not have
routes of their own, and if for some reason they must, those routes do
not supersede routes of their master. This should generally be the
case, but unusual configurations or errant manual or automatic static
route additions may cause trouble.

The highlighted text is the part I'm most curious about. According to this article, the ARP subsystem "matches replies on an interface basis". My question is: Is this necessary?

Is there some reason why the ARP subsystem, or the TCP/IP stack, or the OS in general couldn't just treat all incoming packets the same, regardless of which interface received them, and just "randomly" pick an interface to use when sending any packet, assuming that the interface was physically capable of delivering the packet.

To put it another way, how important is it that a TCP/IP connection "stick" to a particular network device? I see three options:

a) a connection must be "attached" to an interface, and can only use that interface for communication.
b) A connection is "related" to an interface, and the system should "prefer" that interface over all others, or
c) A connection is not related to any particular interface, and packets can be sent to any interface (that can deliver the packet) at any time.

Is there any logical reason why option C could not be implemented at the OS level?

For instance, does the router care if it receives two TCP packets for the same IP address, for the same port, but from different ethernet MAC addresses?

Hopefully, all of this makes sense...

[Brendan]

Hi,

To put it another way, how important is it that a TCP/IP connection "stick" to a particular network device? I see three options:

a) a connection must be "attached" to an interface, and can only use that interface for communication.
b) A connection is "related" to an interface, and the system should "prefer" that interface over all others, or
c) A connection is not related to any particular interface, and packets can be sent to any interface (that can deliver the packet) at any time.

Is there any logical reason why option C could not be implemented at the OS level?

Depending on how you count them, networking is broken up into several layers.

For TCP/IP (and UDP/IP), you can do load balancing and fault tolerance at the "network layer" with IP. The IP forwarding algorithm already supports fault tolerance, and could be adapted to use multiple links to improve bandwidth (e.g. if the routing table has 2 or more links to the same destination/s, then you could do a "round robin" thing to spread load across all of them).

Ethernet bonding happens at a lower level (the "data link layer"). Because it happens at a lower level it can need special support from things like switches; but it can also be more powerful (e.g. a link can fail and nothing in the "network layer" would need to care, even if the network layer is using something like IPX and not using IP at all).


Cheers,

Brendan

[bluemoon]

c) A connection is not related to any particular interface, and packets can be sent to any interface (that can deliver the packet) at any time.
Is there any logical reason why option C could not be implemented at the OS level?

Networking require agreement in communication method at both ends. So if there is no translation layer (eg. a switch or device that handle such method), other "normal" computer may only send packet (including ack) to one of your adapter.

[rdos]

c) A connection is not related to any particular interface, and packets can be sent to any interface (that can deliver the packet) at any time.
Is there any logical reason why option C could not be implemented at the OS level?

Networking require agreement in communication method at both ends. So if there is no translation layer (eg. a switch or device that handle such method), other "normal" computer may only send packet (including ack) to one of your adapter.

The reason is that at the ethernet level packets are delivered to an ethernet address, not an IP address. So when TCP/IP data is sent, it is the destination ethernet address that counts, not the IP address. The translation between IP and ethernet addresses is done with ARP.