tossing large data structures to user threads

[Adek336]

Hi!

I can send large messages, which are only limited by malloc(). However, how would I give the user threads access to the data if the size of the message can vary that much?

• I allocate virtual pages and fill them with a copy of the message. Pros: only the requested message is accessible; Cons: this is space-wasteful if small messages are passed (it takes a whole page anyways), unless the app copies the data to its own heap and requests to dealloc the page struct;
• Firstly make a call to the kernel to request the message's size, allocate a buffer and request the message to be copied into the buffer. Pros: doesn't waste memory because of page-granulity; works quite well; Cons: It would require the infastructure to get the message's size, and some time later, get the SAME message to be copied. Perhaps this is doable;
• Share the kernel structures which contain the message for reading access for the user app. Pros: least memory overhead, least memory movin'/copyin'. This would allow the app to access the data without allocating new buffers. Cons: apart from the message, other data which are near the message, like other messages or book keeping structs are availlable for the thread for reading. Also, there would need to be a call to dispose the message later.
• The thread may as well read the message as if it was a socket, so it would read it in small 128-byte (or any other granulity) bits until it was read completely read. Pros: well not much here; Cons: This would need a complex infrastructure for the sockets, if I wanted to do this properly. Also, the user app would need to read from the socket before it could do anything useful with the message. The first few bytes would contain a header though, so after the first read the message's size would be known.

IPC rulez

Cheers,
Adrian.

[Gnome]

I think the third option is the best one, but put the structures in pages allocated to the target process. Do not keep it in kernel space.

Dedicate a region of virtual memory to IPC pages. You can allocate space in the pages malloc()-style, thus avoiding memory waste for small messages. You will, like you said, need a way to tell the kernel that you're finished using the message, and it can be deallocated.

[BI lazy]

I'd keep small fixed size messages for the kernel to store and forward. That's quick and doesn't cause too much overhead.

For larger message structures either use userland message queues or sockets.

I can also imagine sharing a set of pages between the two or more processes who wish to communicate. Plce some relocatible structure in it (kinda circular buffer) and write and read messages to/from this location with dedicated library functions. With a sound implementation of shared memory (this shan't be a problem for you, adrian), this is the best option.

hth

[Dreamsmith]

[*]Firstly make a call to the kernel to request the message's size, allocate a buffer and request the message to be copied into the buffer. Pros: doesn't waste memory because of page-granulity; works quite well; Cons: It would require the infastructure to get the message's size, and some time later, get the SAME message to be copied. Perhaps this is doable;

How about this: first the app allocates a buffer and requests the message, supplying both a pointer to the buffer and it's size. If the message will fit in the buffer, the call returns successfully. If the message will not fit, the call fails or returns a partial result. In either case, the call returns the actual length of the message, so the buffer may be realloced smaller to save space or larger to satisfy the need, and the call redone with the larger buffer.

In the best case, the message is passed in a single call. In the worst case, two calls are required, one to get the buffer size and one with the buffer expanded to the needed size. So the worst case is the same as what you proposed above, whereas the best case is it all gets done in a single syscall.

Furthermore, since applications usually have a pretty good idea how large incoming messages are going to be, the best case scenario is also the most common scenario.

Either that, or go with the socket interface. The other options require the kernel to intrude into userland memory management -- that's the app's domain...

[Pype.Clicker]

i guess i'd be rather with BI.

- Pick a small fixed-sized message framework that is enough to convey most messages with affordable memory cost. (32/64 bytes would be a maximum)

- Try to keep messages 'local' (e.g. on stack rather than on heap), so that a message-polling loop can always reuse the same buffer for all message, or that the next call to "read message" automatically discards the previous message. If the app needs to keep long-term information, it has to copy it itself.

- When passing large data structures, just place in the small message the information required to identify those data structure (vaddr and size)

Code: Select all

emitter:
    Message template={'inline' part};
    AttachData(&template, offsetof(Message, ld), LargeData, 
                    sizeof(LargeData));
    AttachData(&template, offsetof(Message, ld2), LargeData2,
                     sizeof(LargeData2));
    SendMessage(&template);

kernel 'sees' that LargeData#sizeof(LargeData) should be 'exported' out of the emitter's address space, creates the appropriate shared buffer and writes down a shmem-key in the in-kernel copy of the message.

At reception, the kernel can use something like morecore() to allocate space required for LargeData automatically or let the application request attachments explicitly.

Code: Select all

    Message m;
    GetMessage(&m,WITH_ATTACHMENTS);
    // the kernel automagically allocate addresses and maps them
    // so that LargeData and LargeData2 are there, then fixes    
    // pointers ld and ld2 in m.

    GetMessage(&m,NO_ATTACHMENTS);
    // inspect the message, retrieve ld.key and ld.size
    buf = malloc(m.ld.size);
    CopyAttachment(buf,m.ld.key);
    DiscardAttachment(buf,m.ld2.key);

[Adek336]

Thank you all for your help!

I am now considering Dreamsmith's and BILazy's/Pype.Clicker's designs; both do accomplish the goals. Especially having the message buffer a stack rather than a heap (the stack- a reallocatable buffer on the heap). GetMessage(mailbox *TheOnlyMailboxIWIllBeAcceptingMessagesFrom) would need to memmove a bit from time to time, but imo, this isn't a big problem. I like the way BILazy's/Pype.Clicker's design sends small messages to give large data; (I would need to cow the data, though. Ain't it so?). But when sending messages without use of shared memory, I would use DreamSmith's way of telling if the user buffer is too small.

Thank you again. Cheers ;D I'm going on vacation to Spain tomorrow anyways so you'll hear from me not before two week's time. Bye
Adrian