Implementing network support

[zhiayang]

No, this isn't one of those 'how 2 connect os to internets' posts.
(It might, however, come across as one)

Now I'm on my way to a working NIC driver, specifically for the RTl8139. The thing is, I don't quite understand how this whole 'internet' thing works at the base level.

I get the part where you send a packet to a DNS server with the readable name, where it responds with an IP address, then you send your data to that address (this *is* how it works, right?!)

What I don't get is the lower level. I understand where the different protocols sit on one another, but what's on the lowest level? Documentation on the wiki is a little sparse, so from what I can gather:

1. User provided data (possibly another protocol, like HTTP) is encapsulated in a TCP or UDP packet.
2. The packet is further encapsulated in an IPv4 or IPv6 packet.
3. ???

This is my question, do I just fill in the source and destination IP addresses in the IP packet, then send it off to the NIC to transmit?

Thanks.

[Combuster]

You're missing the ethernet layer. You need a MAC address of a router (the default gateway), and you'll want implement ARP to turn IP addresses to MAC addresses.

[Candy]

The LAN only knows MAC addresses. Your LAN connected devices only know how to talk to other LAN connected devices. They all use MAC addresses to know each other.

So that means what whatever you want to do you have to know: What's the MAC address of the first hop to send it to?

Thankfully, there are some protocols:

- DNS, allows you to translate "www.google.com" into 173.194.65.106.
- Routing logic, allows you to determine that packets sent to 173.194.65.106 should be sent to:

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.2.254 0.0.0.0 UG 0 0 0 wlan0
link-local * 255.255.0.0 U 1000 0 0 wlan0
192.168.2.0 * 255.255.255.0 U 9 0 0 wlan0

- 192.168.2.254's MAC address as first hop on the network, and then sent out on wlan0.
- ARP, allowing you to translate 192.168.2.254 into

Address HWtype HWaddress Flags Mask Iface
192.168.2.254 ether 84:9c:a6:5a:b9:a3 C wlan0

- 84:9c:a6:5a:b9:a3. This you then fill in to the target MAC address, put the IP packet for 173.194.65.106 in that, put the TCP packet into that, put the HTTP GET request for "www.google.com" into that, and send it.

Of course, how does DNS then get its data? Same trick, except you use the dns server that you know to be 192.168.2.254, and you then use ARP to determine its mac address, send a DNS packet, receive its answer... etcetera.

Good luck!

[zhiayang]

Hello, thanks for the replies.

1. So ARP is essentially just another layer below the IP layer?
2. I'm a little confused on how this works.

I'm assuming I first need to send a DHCP request around to get an IP address... Then what?

For a simple thing like sending an ICMP echo packet to a server on some address, what should I do?

I'm thinking
1. Use DHCP (on top of UDP) to get an IP address -> broadcast to MAC FF:FF:FF:FF:FF:FF ?
2. Broadcast using ARP to target IP address, requesting MAC address
3. Send data to the MAC address received?

In particular I'm confused about how the whole local -> global thing translates, I can't be expecting something on the other side of the world to respond to my ARP request, so is this handled by the router or something?

[thepowersgang]

Networking is built up with layers. In home networks (and most enterprise ones too), there's 802.1 ethernet "segments" between machines. These are sometimes referred to as "ethernet broadcast domains", because a broadcast ethernet (MAC address of FF:FF:FF:FF:FF:FF) packet will be delivered to every machine on this segment.

To access outside this segment, there needs to be a computer that's connected to two different ethernet segments, and uses a higher-level protocol to route between these two. The two common "Layer 3" protocols are IPv4 and IPv6.

ARP or ICMPv6 ND (Neighbour Discovery) are used to turn a layer 3 (IP) address into a MAC address (assuming the target L3 address is on the same ethernet segment as you, if not nobody would respond to the request). For L3 addresses outside of your ethernet segment, a routing table is used to figure out what machine is the next hop along the way to the target address (in home situations, this is always your modem, but in more complex networks there might be multiple routes set up for different destination networks).

DHCP(v4/v6) and ICMPv6 RA (Router advertisements) are just used to statelessly assign addresses to client machines on a network, and don't play a role after configuration.

[zhiayang]

So the process is something along the lines of:

1. Check if target IP address is local (192.168.x.x right?)
2. If so, use ARP to get MAC address of target IP address, then send IP packet

3. If not, get IP address of local router
4. Use ARP to get MAC address of router
5. Send the IP packet to the router's MAC address.

Is that so? I'm a bit confused here, with steps 3 and 5 in particular:
a. How do I find out what the local -> outside connection point's IP address/MAC address is? Is it possible to get its MAC address directly and skip step 4?
b. Since I'm sending a packet to the router with a MAC address through ARP, I use that packet to encapsulate the actual IP packet that contains the real target IP address?

Thanks for your responses, this is really really confusing...


EDIT: thePowersGang, you mentioned something about a 'routing table' in your reply that is used to get the address of the router... What exactly is this table, and how do I get it?

[Combuster]

(for typical IPv4)
1: Use DHCP requesting network information (using the broadcast MAC and IP)
2: Store your IP, the gateway's IP, and the netmask. (and all the other things)

For any packet sent over the network:
3: Check if the destination IP is on the same network (dest_IP & netmask == your_IP & netmask)

4a: If it is, retrieve the destination MAC address using ARP (when needed)
4b: send the packet to that mac address

5a: If it isn't: retrieve the MAC address of the gateway using ARP (when needed)
5b: Send the packet to that MAC address. The gateway will forward the packet as it sees fit.


Local isn't guaranteed to be 192.168.x.x, as there are several such ranges and you might need a router (and therefore, a gateway IP) to reach other parts of the local address space.

[thepowersgang]

Combusters reply is probably more understandable than mine

In response to the routing table question - Most OSes don't just assume that they'll be used as a client machine (On one network only, and with just a local subnet and a default gateway), instead they have a "routing table" that is used to determine where to send packets, consting of subnets and gateway addresses. The networking stack checks the destination address against this table and uses the most accurate match to determine where to send the packet. A normal user machine has only two entries in the routing table - One matching against the local subnet (with a next hop address of 0, indicating send directly), and one matching every address (with a next hop of the local router).

[Combuster]

And a third for 127.0.0.1

[Candy]

Attempting to make it as explicit as I can...

> For a simple thing like sending an ICMP echo packet to a server on some address, what should I do?

Start your networking driver & stack (ie, allocate buffers). Run DHCP and using that, get an IP address for yourself, your gateway address, the netmask and the DNS server IP. Store all of them. In a more advanced form of your OS you'll use this to update the routing table, but right now we're just going to use this as the only connection.

When trying to ping the server, first determine how to send it a message.

If the IP address AND 255.0.0.0 is equal to 127.0.0.0, then it's to yourself. Send back immediately.
If the IP address AND your netmask is equal to your own IP address AND your netmask, then it's an address on this link. Use it directly for the next step.
Else, replace the address with the gateway IP address.

Now, take that IP address and send out an ARP broadcast message - it has 4 fields, "my IP", "my MAC", "your IP", "your MAC". Fill in broadcast for "your MAC", the requested IP for "your IP", your own IP for "my IP" and your own MAC for "my MAC". You'll get a reply in the exact same format with "my" and "your" swapped around, and "your MAC" (which is now in my MAC) filled in with the correct target.

Then, send all packets you want to send to that host to the MAC you now found out.


Advice: Reboot your computer into 100% fresh state, start Wireshark on your network port and surf to google.com. When the page has loaded, stop Wireshark and look at every packet it's seen - they're all relevant. And yes, that's a few hundred packets. No filtering.