Emulating real mode to run VESA calls?

[Artlav]

Imagine a 8086 emulator, which have the system's memory as it's RAM and functions to pass through the IO ports.
Load it up with glue code that would execute a call to VBE, and you can get VESA to work from long mode.

How feasible is it?
Was it done before?
I distinctly remember seeing an x86 emulator library made for that very purpose 5-10 years ago, but can't google it up now.

Tried it myself, with an 80286 emulator i made a while ago, and it kind of works.
That is, in Qemu and in Bochs it works perfectly, but on real hardware it hits an opsize: prefix early on and goes nowhere.

Is that the end of it, needing a full-size 386 instruction set emulation, or are there some limits to what instructions can be used in there?

[Unsigned]

If what you see is a problem related to the inability to interpret a 386 instruction, then I suggest to use an emulator for a more recent processor, like the 386 instead of a 286. I don't know if there are limits on the instruction set, but the 386 is so old that the video ROM programmers may just have assumed the processor will be already a 386 or better. If this works with Bochs and Qemu it may be because their video ROM BIOS was designed to adapt to several conditions, for example, in case the user wants to emulate an 8086 computer. But code in actual graphics cards may assume a better processor, mainly VLB/PCI/AGP/PCIE cards (and not ISA ones), since if those buses are available it means there's a newer processor. You may want to emulate even newer instruction sets, just in case a card requires, for example, CMOV, which is available since the Pentium Pro. I encourage you to do so, since having VESA support will improve the user experience by providing access to higer resolution and color depth graphics modes. Without it, you'll need to rely on drivers for specific cards, which not only will be a lot, but also difficult to find information for, and difficult to test if you don't have a specific card.

[Artlav]

If what you see is a problem related to the inability to interpret a 386 instruction, then I suggest to use an emulator for a more recent processor, like the 386 instead of a 286.

Yup.
The question is - is there an emulator already out there which is fit for that purpose?
Butchering Qemu or Bochs isn't going to be particularly easy, nor would it be easy to write a full 386 instruction set emulator.

[Combuster]

libx86emu:

Code: Select all

/****************************************************************************
*
*                                               Realmode X86 Emulator Library
*
*               Copyright (C) 1996-1999 SciTech Software, Inc.
*                                    Copyright (C) David Mosberger-Tang
*                                          Copyright (C) 1999 Egbert Eich

I think it was somewhere in x.org

[Octocontrabass]

How feasible is it?
Was it done before?

Windows does it.

Is that the end of it, needing a full-size 386 instruction set emulation, or are there some limits to what instructions can be used in there?

Windows seems to implement a 486-like subset of x86, so you're unlikely to see anything other than that.

[Artlav]

I think it was somewhere in x.org

Found it: http://www.liwjatan.at/files/src/hwinfo/
Looks interesting, thanks.
And i was right, just the opsize-like prefixes essentially require doubling most of the code, no easy way to turn a 286 emulator into a 386 one.