Network Subsystem

[core-ix]

Hello,

I'm newbie OS developer and I'm interested especially in networking. My current project include writing network subsystem for microkernel OS. The main IPC provided by the OS is message passing. My question is how to organize the network subsystem. I know how it's done in QNX, Minix. I have strong knowledge of the BSD network stack. I saw how it's done in NewOS ... and some others too

But can you share some thoughts? What do you think is the best way to do it, without loosing speed or functionallity. Besides complexity of the code I think this is most concern - make it fast (but hard to change) or make it extremely flexible (but probably slow).

Thank you in advance.

[Pype.Clicker]

well, one of the abstraction that imho worth investigation is the 'packet filters' -- and more specifically, the ability to tell the network subsystem that any packet that match a specific pattern (such as, being an IPv4 packet with protocol type=6, source address=12.34.56.78 and tcp.dstport=80 ... should be delivered to a specific 'file handle').

From this, you could build a whole subsystem more or less freely, for instance loading an ICMP kernel module or running an ICMP server application without having the microkernel noticing the difference.

What's probably the most important in a microkernel environment will be the support of "packet buffers" that can be passed efficiently accross execution contexts (kernel, server, applications, etc) without requiring copies, etc.

[distantvoices]

I'm considering to implement a kind of packet filtering subsystem inside the IP-Layer - it is to reside on a level between tcp and ip - it looks into the routing table, it has to look into its own rule set (which I intend to be a (double) linked list of rules), it has to check tcp pcb's if that is required by any rule. First rule match -> there we go.

Second: this passing network packet buffers around without too much copying: especially the tcp sent list will be hard to do without copying - we do need to copy yet to be ack'd data from the resend buffer to a dedicated packet in the offset/size of not ack'd congestion window) but on the other hand - I do agree. If we could pass shared memory entities around for big data, it'd be quite nice. Except the fact that we have to split the data into packets which then are to be extended with network header stuff /deprive packets of network stuff and merge it ...

Currently - as with my micro kernel - I'm yet learning to walk in the area of network programming, so what talk I in big tones. *smirk* But feel free to bug me.

F. ex. the nic drivers - I'm currently including a few into the net service and offer a possibility to request that the kernel sends us the irq messages so we can take care of our device.

F. ex.II: you will find an object oriented approach for the interface layer the nicest to implement, for the interfaces have to provide a set of functions - have them fill in a struct, insert the struct into a linked list ... voila ...

Stay safe

[Pype.Clicker]

note that it should be possible to implement "packet filters" as a single "state automaton" (or more likely, a decision tree) rather than a list of rules, meaning that you can aggregate similar prefixes and share tests. New rules added at runtime add edges in the automaton and possibly accepting states.

[distantvoices]

I'm not sure if I understand you completely, Pype.

You mean a state graph?

Well, I need to consider this more of a bit ere I respond in full length.

What troubles me too is this Patricia Tree stuff in the BSD-Network Stack routing tables. I don't get it: how do they resolve the relation address:route:interface? Are the leaves of the search tree linked with some nodes in a list? Or de the leaves themselves represent the route entries and keep pointers to the responsible interfaces?

Well - lotsa stuff to think about. I'm gonna flesh out the linkage between interface layer and ip layer. At least the loopback thing is already there, so let's go on and test some of the protocol stuff. I want to see packets run around after all these months of drawing and developing and code churning.

[core-ix]

well, one of the abstraction that imho worth investigation is the 'packet filters' -- and more specifically, the ability to tell the network subsystem that any packet that match a specific pattern (such as, being an IPv4 packet with protocol type=6, source address=12.34.56.78 and tcp.dstport=80 ... should be delivered to a specific 'file handle').

In the context of microkernel I plan to make each OSI layer separate process and a network manager to coordinate them all. My idea is: upon registering a new interface (when the network driver is loaded) user supports (as a command line parameters) what protocols should be loaded for that interface. In the network manager we have a struct of linked lists with communication points to the layers. When packet is processed we simply pass it to each (or some) layer (identified by a process).

Somehing like this:

Code: Select all

/* Per Interface struct -> layers from the OSI model */
struct _per_if_struct {
    L1 *l1;
    filter *pre_l2;
    L2 *l2;
    filtet *pre_l3;
    L3 *l3;
    filter *pre_l4;
    L4 *l4;
    filter *pre_l7;
};

for each layer entry we have ID which determines what process to send the data to:

Code: Select all

/* Layer 3 struct -> IP, IPX, others ... */
typedef struct _L3 {
    int ID;
    object_t comm_point;
    /* other fields */
} L3;

Of course it is not that simple. I want to implement policy based routing, which shoud reside in the network manage ... but I think this model is quite flexible - it is easy to load different filters, converters, markers and even layers in runtime, you have per interface filters, which means easy/logical configuration, easy way to create vlans, tunnels, fast protocol modifications ... since they're user space processes.

What's probably the most important in a microkernel environment will be the support of "packet buffers" that can be passed efficiently accross execution contexts (kernel, server, applications, etc) without requiring copies, etc.

Well, this really is the *PROBLEM* so far But avoiding passing data here and there sacrifices flexibility ...

[core-ix]

What troubles me too is this Patricia Tree stuff in the BSD-Network Stack routing tables. I don't get it: how do they resolve the relation address:route:interface? Are the leaves of the search tree linked with some nodes in a list? Or de the leaves themselves represent the route entries and keep pointers to the responsible interfaces?

Look at this link for more info: http://unilans.net/ivo/BOOKS/ ::)

[Colonel Kernel]

In the context of microkernel I plan to make each OSI layer separate process and a network manager to coordinate them all.

I see two potential problems with this: No one uses the OSI model (except as a conceptual tool), and performance will suffer greatly if you have many different process switches involved in the processing of each send/receive.

It would be a good idea to just stick with TCP/IP, since it actually works and is used everywhere. In terms of the architecture, I would instead have a separate process for each NIC driver, and load the protocol stacks as shared libraries into each process (i.e. -- NIC) that needs them. That way the only process hop is between the requestor and the network stack for a particular interface. If your microkernel support IPC based on page-flipping, you can avoid copying almost entirely (you can even do the same with shared memory, although this is a bit more clunky IMO).

Well, this really is the *PROBLEM* so far But avoiding passing data here and there sacrifices flexibility ...

Not if you use shared libraries (see above). Actually, your proposed approach would be a really good idea on an architecture like Singularity, where the cost of a process switch is negligible and dynamically-loaded shared libraries are a no-no anyway... However, for us mere mortals without our own pet compiler and language projects, this remains out of reach.

[Pype.Clicker]

In the context of microkernel I plan to make each OSI layer separate process and a network manager to coordinate them all. My idea is: upon registering a new interface (when the network driver is loaded) user supports (as a command line parameters) what protocols should be loaded for that interface. In the network manager we have a struct of linked lists with communication points to the layers. When packet is processed we simply pass it to each (or some) layer (identified by a process).

well, you're of course free to follow that way, but i wouldn't recommend it. For speed purposes (and yes, you always come to speed issues when dealing with networks), you wouldn't like the TCP/IP packets carrying the 4GB of DVD from FutureOs distro to be slowed down due to invoking 'NIC server', then slowed down again by NIC-to-IP messaging, then again by IP-to-TCP messaging and finally by TCP-to-app data delivery. Instead, you might want to have TCP/IP a library and several 'network drivers', each one bound to a given NIC device and implementing TCP/IP stuff, then delivering stuff to the app.

Yet, that wouldn't quite be the way i would try to follow. I'd rather try to have NIC drivers at kernel level (at least, enough code at kernel level so that an incoming packet can be retrieved on main memory and a 'responsible component' gets notified). The 'responsible component' would be defined by looking at packets headers, matching the packet against 'filters' that acts as 'guardian' for the packet. For instance, when you request a TCP socket with megatokyo.com:80, the network server would first resolve the name into some IP address, initiate the connection (e.g. hooking "syn+ack" packets coming back from megatokyo) and then sets up a new filter that will make packets belonging to that TCP connection to be delivered straight to the application.

Yet, with that model, it would be up to application software to send requested duplicatas ... so you still might want to have the network server to handle TCP for you instead and only deliver payload bytes.

As you can see, there's no loss of flexibility here: all depends on how you set up those 'filters'. Another server could for instance receive all the ICMP traffic, etc.

When i said "avoiding to copy the data here and there" i meant that: copying, not 'passing'. If you manage to have your data in a neat "packet buffer" so that you can write new headers when needed, etc. you could very well just write data when they arrive at a NIC, and then just pass along the "descriptor" of the packet buffer to the different components involved (TCP/IP stack, application code, packet filters, network monitoring, routing tables ... ) without doing additional copies. When considering the NIC bandwidth and the memory bandwidth, it will be decisive, especially if you plan to build a router from your OS.

[distantvoices]

You 'd have to take the data out of the NIC Receive ringbuffer anyway, else it'd run out of buffers for receiving frames.

I'd keep the nic drivers as subsystems inside the net service. One can give them adequate priority.

The more, I'd avoid splitting the layers into separate processes - keep them in one process. You can spawn some threads to take care of net events, timeouts and fragment killing f. ex.

Stay safe

[0Scoder]

My OS works on the concept of streams for IPC (areas of memory shared by the proccesses involved, with the protocol managed by common libraries (although perhaps there will be inter-computer (network) streams later to allow parralell operation)).

Thus, protocol layers can be linked to each other in any way the user wishes (i.e for sending packets). Also a packet filter can be used to 'tap into' a stream, and thus can route packets back to their applications (also through a stream). Interpretation of packets can be done using shared libraries to prevent needing to rewrite this code for both the filter and the handlers.

The idea of this is to reduce messaging overhead to almost nill (no copying needed). Also various things (like IDS) can just silently link into the stream recieveing raw packets, without any nuisance for packet drivers.

But would it work in practise?......