IPC: Streaming data across processes

[Pype.Clicker]

Hey, dudes. I'm at the point of implementing something like inter-process data streaming. I have a reference design sketched and starting implementation tests before i go further to userlevel invokation and wrapping libraries.

I just noted spoon kernel (iirc) has a mpeg player on its latest demo and i was wondering how many of you already have a kernel that features such data streaming (well, assuming that spoon isn't just rendering each frame directly on the screen buffer, it should need some way of streaming data along process, right ?)

and if you're doing something like that, how important was it for you to avoid "unnecessary" data copies ... or do you have something that can collate several small data chunks into a single datachunk later, etc.

[distantvoices]

re spoon: iirc, durand renders frame by frame to the dedicated offscreen buffer for that window, which the app shares with the gui server of spoon. the spoon gui server then simply blits the buffer to screen - clipping included. So, that's shared memory, and for videos it works pretty well. I've tried playing three times the mpeg demo in spoon. It played 'em back a bit slow, but it worked.

Well - for this kind of stuff I personally would give the playback app access to the frame buffer, communicate limits and clips and voila, have it do the drawing client sided directly on the vram. But that's a different story.

I'm currently copying data between processes with a dedicated system call: user_copy(src proc,target proc,src buf, target buf,len). It can be of any length, pipes data throu in chunks of max 4096 mb and I'd prefer a shared memory approach to this ugly beast - but till now it has served me pretty well. *roar*<-thats the ugly beast.

Gonna look at your draft now... <i>ready for freddy</i>

Just one question: no. two.

1. can the receiving process access the pages mapped in by the send-message call? You've stated somewhere in the teacup, that the ain't accessible from userspace. its this one:

The stream is built of two component: a channel and windows. The channel is the collection of pages that is used for communication and it should be present in some memory shared by the two process that use the stream. It is also invisible to UserLevel.

I'm not shure if I understand correctly: you then intend to have a possibility to map pages of a channel (that shared mem area) into user space - visible to the application? Is that correct? You further intend to make it possible to discard those userland accessible mappings after usage? Hm. Why not keeping them? Wouldn't that save some mapping operations?

2. Do these pages remain in both process' address space for further availability? Say to pipe throu data of more than 8192 bytes, where window size == 8192 - this would need two successive write-reads.

3. The reader can do with the received data what he wants, right? Even map out the page to somewhere else and instead give back another, empty one (oh sweetness of paging)? Does the writer get this newly mapped page too? (need to reread that stuff ...)

Oh, I've lied. It's three questions actually. Shame on me.

PS: Non, J'ai pas eu d'occasion d'aller au cine pour regarder H2G2. C'est dommage - mais a'l autre main, j'ai passee une vacation (est ce que c'est majuscule ou feminin?) joyeux en Ober?sterreich.

[Pype.Clicker]

1. Actually, the 'channel' is just a logical structure controlled by the kernel. It tells both process which are the physical frames involved in the stream, which of them are currently mapped by the reader and by the writer. That's just a small structure along with an array of frame numbers -- not the frames themselves (i may have failed to express that clearly).

Since both process will use it to coordinate their access, it has to be in kernel-level "shared" memory, so that the user code don't (for instance) mess with what physical frames are involved.

You further intend to make it possible to discard those userland accessible mappings after usage? Hm.

Well, the idea was to keep the window so that the userlevel ignores wrap arounds: if you mapped half your window and then want to write a "message" that is over half of the window, the discard-and-prepare approach ensure that you've another fresh window to write the stuff. That's a bit like a "magic buffer" that can transmit its content to the other side and then appear to have fresh space again when the call returns.

Why not keeping them? Wouldn't that save some mapping operations?

well, i want to avoid the case where the reader can't tell whether the data might or might not change while it's processing them. For security and sanity reasons.

2. Do these pages remain in both process' address space for further availability? Say to pipe throu data of more than 8192 bytes, where window size == 8192 - this would need two successive write-reads.

You cannot indeed send more than your window's size in one system call. However, if you have e.g. twice the window to be send, you don't necessarily need to wait for the reader if the channel is large enough: you prepare the window (e.g. with 64K), you write data, you submit those 64K and prepare another 64K (that'll all done in one system call), you now write the additionnal 64K and submit it. And voil?, you've been sending 128K through a window of 64K.

Of course if the reader isn't reading fast enough, you might need to wait for additionnal pages before "prepare another 64K" step ends.

3. The reader can do with the received data what he wants, right? Even map out the page to somewhere else and instead give back another, empty one (oh sweetness of paging)? Does the writer get this newly mapped page too? (need to reread that stuff ...)

Well, the idea is to have two flavour: the reader can either leave the pages on the channel (that is, read it, dismiss it) or it can take it for its own purpose against free pages of its own. In that later case, it can indeed then use those pages on another channel to another process -- optionnally after having altered their content.

[durand]

re spoon: iirc, durand renders frame by frame to the dedicated offscreen buffer for that window, which the app shares with the gui server of spoon. the spoon gui server then simply blits the buffer to screen - clipping included. So, that's shared memory, and for videos it works pretty well. I've tried playing three times the mpeg demo in spoon. It played 'em back a bit slow, but it worked.

Correct. I'm using messaging (memory copying) through a very long path: mpegplayer -> vfs -> device driver -> vfs -> mpegplayer -> shared memory to GUI server -> sync signal -> gui copies shared memory to screen. And yet it's still running at a decent speed..

A nice IPC method for streaming could be using shared memory and a signalling technique. You could implement read-write pipes with zero-copy if you do it smartly.

Create a shared memory region with a controlling struct inside (bytes in buffer 1, bytes in buffer 2, buffer 1, buffer 2, a few locks) and your one process can write into buffer 1 and the other can read from buffer 1. And vice-versa for buffer 2. If you don't want to do polling, send a signal to indicate that data is waiting.

So you get zero-copy of large amounts of data with userland flow control and perhaps a single IPC signal to communicate that there's waiting data. It's very cheap and fast. And even if you do without the zero-copy (cos it's a little tricky), it's only a matter of 2 userland memcopies into and out of the buffer - removing the kernel copy & store & deliver copy.

Still much faster.

For security, add in some limits, bounds, pid, tid and data integrity checking. Then your userland apps can send rubbish data and your servers are cool. They just return "unknown command or bad data." errors all the time.

[Candy]

In-kernel I'm still planning on using a sort of AmigaOS-inspired mechanism. The lowest level driver that cares (IE, has a say about some limits) allocates a buffer to hold data and passes it on. No memory or buffer may be allocated above, but the buffer can be passed to userspace. Each read instantly allocates a buffer for it, each write frees it too. You can allocate a buffer for yourself for use with a given device although I'm not sure how to accomodate this for some devices. I might need a single copy.

Then again, you'll only be passing the pointer to the page around, speeding up all IPC.

Oh, almost forgot, it completely and totally breaks all normal POSIX compliance. You'd need to emulate it. But then again, I don't care. My reads with the posix emulation are going to be exactly as fast as posix originally was (or at least approach it).

[Pype.Clicker]

IThe lowest level driver that cares (IE, has a say about some limits) allocates a buffer to hold data and passes it on. No memory or buffer may be allocated above, but the buffer can be passed to userspace.

Yep, we do find that kind of design in several I/O-bound OSes such as router-specialized OSes: the 'network card driver' that receives a packet allocates memory for the packet and the network driver that completes delivery of a packet returns the memory to the source.

The problem is for protocols that have variable-sized data chunks. I think of b-encoded data or whatever type-length-value encoding where you usually issue a read of constant size to know what's the size of the next chunk and _then_ read that out. You also get similar issues when trying to create such streams: you'll write the size then write the data.

[Candy]

The problem is for protocols that have variable-sized data chunks. I think of b-encoded data or whatever type-length-value encoding where you usually issue a read of constant size to know what's the size of the next chunk and _then_ read that out. You also get similar issues when trying to create such streams: you'll write the size then write the data.

That can be handled with a simple struct:

Code: Select all

struct object {
    uint32 length;
    char content[1];
}

which you overlay the actual data. Then,

Code: Select all

memcpy(dest, object->content, object->length);

which looks pretty darn intuitive suffices.

[DruG5t0r3]

I'm into the design process of IPC...and well the only way I thaught of it possible since all processes have their own context is to send a syscall to the kernel so that it fetches whichever area of memory from process x and copies it to process y memory space. This involves quite of lot of fuss such as switch to CPL0 and loading quite a few pages etc....is there any better way of doing it?