Also, how can I access and enable VESA modes from my C kernel (without a vm)
C + protected mode + no vm + vesa eliminates pretty much all options without considerable amounts of hacking/work :-/
the only clean option:
4: write an emulator
Given that, you might consider lowering your standards and either
1: leave protected mode, call int, return (which requires a decent amount asm, violating the C premise)
2: virtual 8086 mode (violating the vm premise)
3: VBE protected mode interface (requires 16-bit pm code, assembly once again, also IIRC its poorly supported)
5: write specific drivers (violating the VESA premise, since at this point it becomes a native mode instead of a VESA mode, limited support, since high-resolutions are nonstandard and a VGA doesnt provide those without hurting your eyes)
Where 1 and 2 are the most common solutions, 1 being the easiest, and 2 being technically better.
what's this whole 'plane switching' idea?
A standard video mode only provides access to the framebuffer from 0xA0000-0xAFFFF (64k). Many resolutions don't fit in that. Vesa and VGA provide methods to access all of their memory through that area using bank switching/plane switching respectively. What you do is tell the video card to map Axxxxh to location y*xxxx+z in its framebuffer. Changing y and z are considered bank/plane switches.
for VESA cards you can simply do an int 0x10 call, and the card will change its 'z' (y is always 1)
for VGA cards you have to write to the vga ports to achieve the same thing. However, on the vga y will generally equal 4 and z can only be 0..3, while on a VESA z can contain very large values.
Since this approach can be troublesome, many modern video cards use a linear framebuffer: the location you write to becomes something like 0xE0000000 + location, which is in such a location that its not hampered by the 64k limit from ancient times.