OSDev.org

Amongst other things, you would want to prioritise responsiveness over throughput to some degree. The latency of response to user input is very significant when playing a game - much more so than the rendering of a single frame being partially incomplete, for example. In server usage, absolute throughput is much more important - especially as the latency of the external connection (i.e. the internet) is liable to be larger than that of the server itself.

I've had a similar idea a long time ago. My idea was more like a game library/framework that ran on bare metal rather than requiring an operating system. However, if I did this, I wouldn't build a high level library that deals with scene and material management because this sort of stuff can be tied and optimised to games and their genres (for example, streaming the scene for a procedural infinite universe vs. a 3d room-by-room shooter), and there is always new academic research done in these fields. I'd instead go for low-to-mid level, such as providing polygonal drawing routines, texture compression, sound playing routines, input routines, etc. such as XNA has done for the X-Box 360.

Unless you're willing to supply device drivers for every kind of device, it would be a nice idea to allow loadable drivers. For example, someone can bundle the game, their NVidia driver, their sound driver onto a bootable CD. You could sell games on bootable CDs if you could allow people to download their driver from online (if you bundle network drivers) and cache it on their hard drive.