I am not familiar with the interface and have never programmed it, but I investigated a little and can share what I found.
First, as you may know, the clipboard function uses guest additions, which are drivers/services that communicate between the guest OS and the host application. Apparently, there are multiple
communication models, and the one used by the clipboard addition is called HGCM (Host-Guest Communication Manager). This model uses port io through a pseudo pci device (the first port in the device's I/O space apparently) to communicate with the host, with vendor id 0x80ee, and device id 0xcafe. You can check how the device is configured inside the VirtualBox sources
here. This model implements IPC of sorts, which connects host "service" modules with the guest additions. The service modules are briefly mentioned in the
Oracle VM VirtualBox Programming Guide and Reference ("7.4 HGCM Service Implementation"), but it is not nearly enough for your purposes. You can see the clipboard service implementation
here. The guest addition on Windows is implemented as a system tray application that registers a clipboard listener. The communication with the host jumps through certain hoops, because it opens a device file which implements the IPC, which then in turn communicates with the host through the pseudo-device's port. Most of the code on the guest side eventually is serialized through those two sources in the respective guest kernel:
VBoxGuestR0LibHGCMInternal.cpp ->
VBoxGuestR0LibGenericRequest.cpp. The former implements the HGCM functions more or less, so if you are capable to reverse engineer it, and reverse engineer the clipboard service implementation, you might be able to get the clipboard integration working. This is all I could find, sorry.