If you haven't done so already, I recommend that you read the wiki page on
Virtual 8086 Mode, though you should be aware that the page only provides an overview of the process; there are too many details which will be dependent on how your OS organizes things like interrupt handlers, process scheduling, and inter-process communication.
Note that these are all things that your OS must have in place
before you will be able to set up V86 support, as my understanding is that the kernel must be able to (at minimum) manage the process(es) which run in V86 mode, trap any system instructions or illegal opcodes which get run in V86 mode for appropriate processing (which includes the INT instruction itself), and pass the results of any V86 operations to the kernel or other processes.
It is the considered opinion of several members here, including myself, that the work needed to get V86 mode running to the point where you can use the Legacy BIOS routines in a consistent and stable manner is more than the work of implementing those functions yourself in protected mode. It should also be considered that, as I mentioned in another thread recently, Intel has already given an End-Of-Life roadmap for UEFI Compatibility Mode Services (with a goal date of 2020), meaning that in a few years support for the Legacy BIOS routines will be removed entirely from newer Intel firmware, and there is every reason to believe that other UEFI implementations for x86 will follow suit.
The big exception to this is video mode setting, for which there is no stable hardware-level standard that can be used for all PC hardware. This is why UEFI boot loaders, and most 'serious' boot loaders for Legacy BIOS systems (such as GRUB), provide a way to set the video mode to what you want before loading the kernel. However, there is more to the story than this.
While the wiki claims that using VBE BIOS routines is the only option for mode-setting, this is not
quite true; what it really means is that it is the only way to do it if you don't have details of the graphics system's hardware and internal firmware. In the days when the VESA VBE standard was set, this would have primarily meant the details of the video registers, but today, it means knowing how to interface with the GPU. If you know which video hardware is being used, and if you have the documentation needed to interface with the GPU, you can write a driver which can set the video mode as needed.
Unfortunately, those are two very big ifs.
The former presents a bootstrapping problem, as you will need to not only determine which driver you will need to use for the current hardware configuration when you install the system (or at least the video drivers), you will need to check for changes to it each and every time the system reboots. The usual solution is to have have a default video mode set by the boot loader, then when the kernel initializes the drivers (regardless of whether they are part of the kernel or separate user-space processes), the driver needs to confirm that the last known configuration still holds, and if not (e.g., if the owner replaced or upgraded the video card), then it needs to go at minimum inform the kernel that it cannot proceed with the driver previously in use.
The latter problem, on the other hand, may prove more stubborn, as not all manufacturers give out the details needed to write a complete driver (e.g., NVidia, Broadcomm), and some which do still hold back on the details of accessing the systems advanced or premium features. While the necessary details have often been partially or wholly reverse-engineered, the manufacturers are also constantly updating their designs to stay one step ahead of FOSS driver groups such as Nouveau. Also, most of the details of the FOSS implementations are really only found in the driver code itself, so if you are implementing (for example) an NVidia driver, you may need to grovel over a lot of that code to see how to adapt it to your OS.
