Octacone wrote:
1.Write a Virtual 8086 emulator. Relatively easy to do, works really well, can "coexist" with protected mode, has paging. Not supported on x64. My personal choice.
V86 isn't an emulator! It's a CPU "mode" (and one that's actually not much different from normal Ring-3 Pmode; just some differences in how the segment registers work, interrupt handling, etc.).
Octacone wrote:
2.Write/port an x86 emulator. Quite a task. Works on x64 doe.
I've done this (port; not write). I had it working in an afternoon. The "libx86emu" emulator is very easy to port, was designed specifically (by SciTech) for running video BIOS code and has been used as such in multiple well-known OSs. Even Windows uses this method (probably not this code; although I won't be particularly surprised if MS had licenced SciTech code at some point) to access VBE on 64-bit systems. SciTech were
the experts on graphics driver development in the 1990s-early 2000s, producing loadable VBE-compatible DOS TSRs for cards without (good) native VBE support, comprehensive sets of graphics drivers for non-mainstream OSs (particularly OS/2), etc. It's a shame that their one-time pledge to release their software as Open Source never came about.
Octacone wrote:
3.Write a dedicated GPU driver. Not quite doable, no publicly available specs, hard to write even if you had them. (except Intel HD graphics).
Intel isn't the only documented GPU out there... Also, if you're only interested in 2D/framebuffer graphics (that's all you'll get from VBE anyway), both the X.org project and the Linux kernel ("fbdev" drivers) have reasonably understandable (and relatively portable) drivers for a huge selection of video devices. There's also the (somewhat dated) SVGAlib collection.
Octacone wrote:
4.Drop to real mode and do your business. Simple but complicated at the same time. You would have to disable paging, be limited to only 1 MiB of RAM, quite a challenge. Not my favorite.
I also wouldn't recommend doing this; it's fairly reasonable if you do it once during startup (at which point you may as well do it before switching to pmode or via your bootloader), but switching to real-mode at runtime risks losing interrupts, which can cause significant problems with things like networking, DMA-based I/O, etc.