Since I'm also interested in (eventually) having VM mouse integration in my OS, I did a bit of searching.
This mailing list post is probably the best place to start; it's a reply from a VirtualBox developer to one of the Syllable OS developers detailing how the guest additions work (there's nothing Syllable-specific there).
Basically, there's a PCI device (VMMDev, PCI device ID 80EE:CAFE) that provides a whole bunch of services; everything from time synchronisation to folder sharing goes through it. Since VirtualBox is open source, you can read through the existing additions and the VirtualBox-side code to get an idea how it all works. It is however fairly complex on a typical OS; there are multiple layers that each request goes through making it hard to understand.
The only independently-developed (and by far the simplest) "guest additions" I've found
those included in MINIX v3. Their code only supports time synchronisation, but does show the basics of how to communicate with the PCI device. It shouldn't be too difficult to use this code as a guide to write a driver that supports mouse integration.