Is this possible?

[Ratty]

Hello,

Firstly I would like to tell you that I have no real experience with assembly but that I am a fully qualified C# developer (MCTS: Web, Windows & Distributed Applications). I have no delusions of building a better-than-Windows OS even having something visible this year. With that said I would like to ask your opinion on the following:

-- 1 --
Firstly I would like to make my own bootloader that will provide options for which drive or partition to bootfrom. This, I know, is simple compared to developing an OS but this is where I'm going to start - entirely in assembly once I have finished building my IDE in C#.
-- /1 --

-- 2 --
Ultimately I would like to build an OS that is, essentially, a system of managing virtual environments so that I may load multiple client operating systems easily. This would be a configurable virtualisation system providing all of the same virtual hardware options as MSVPC or Bochs, ESXI etc... As an educational note for me I intend to come up with my own file system that is not based on any existing version of FAT.
-- /2 --


I intend to do this only as an interesting hobby and only while it is entertaining to me. Sadly I do find things like this enjoyable so I'll have some spare time to put into this. I'm just wondering what difficulties you think I will encounter with creating this kind of 'OS' that is basically a virtualisation platform for other OS (I realise things like this already exist and that learning assembly is likely to be my biggest challenge).

[gravaera]

The idea of an OS that context switched between other OSs on a running machine is interesting, but a few pointers:

1. The OSs running ontop of yours assume that they have access to all system resources, and will treat the entire environment as if you're not there.

2. When a client OS takes control of the CPU, how do you intend to regain control of the system if need be?

3. Switching between multiple OSs on any system with only one CPU is impossible, and unless there is Hardware support for something akin to the originally intended function of the SysRQ key on your Keyboard, then even with more than one CPU, it's still impossible.

3. Switching between multiple OSs, even if it were to be implemented, would take about...5 mins of waiting: (Assuming you managed to jack up some next gen PC Platform CPU from about...oh 10 years in the future)

Why? I assume that each OS would have installed itself into one part of memory or other. Your OS, when it manages to regain control (however) will need to swap the old OS's memory data out, and all of its running programs, and then also RESET all hardware to the state the OTHER OS needs them to be in, before eventually copying in all memory, and CPU/Reg values into for the other OS.

However, if you are extremely l337, you could create a huge emulator, that can run multiple OSs side by side, and that also would have its problems. Most notably speed.

BUT: if you DO manage you create an OS like that on the an architecture it was typically not intended to be supported on, then I suppose your name WILL go down in history, since this IS a great feat.

Nice idea.

All the best
-gravaera

[Brendan]

Hi,

3. Switching between multiple OSs on any system with only one CPU is impossible, and unless there is Hardware support for something akin to the originally intended function of the SysRQ key on your Keyboard, then even with more than one CPU, it's still impossible.

I'd assume the idea would be to (pretend) that many virtual machines were running at the same time, in the same way that a multi-tasking OS pretends that many tasks are running at the same time - mostly by switching the real CPU/s between virtual machines quickly enough to create the illusion that they're all running at the same time.

3. Switching between multiple OSs, even if it were to be implemented, would take about...5 mins of waiting: (Assuming you managed to jack up some next gen PC Platform CPU from about...oh 10 years in the future)

Imagine an OS like Windows, where the user starts Bochs and while Bochs is running they start another copy of Bochs. Now they've got 2 virtual machines running at the same time. No big problem, especially for modern computers (with multiple CPUs). Now rip out every part of the OS that Bochs doesn't need.

Why? I assume that each OS would have installed itself into one part of memory or other. Your OS, when it manages to regain control (however) will need to swap the old OS's memory data out, and all of its running programs, and then also RESET all hardware to the state the OTHER OS needs them to be in, before eventually copying in all memory, and CPU/Reg values into for the other OS.

I'm not sure you understand that (typically) a virtual machine is *virtual*. Lot's of modern OS's using paging to create virtual address spaces - there's no reason virtual machines couldn't also use virtual address spaces.


Cheers,

Brendan

[Ratty]

Thanks Brendan, you seem to understand what I meant. Simply an OS that will run multiple other OSs within itself in the same way as you would open multiple Microsoft Virtual PC applications in Windows.

[Kitsune]

Actually, it (pt 2, not 1) sounds an aweful lot like Xen. Xen can either use paravirtualization (i.e. generally requires software modifications), or take advantage of VT (and AMD's version) on newer processors to do hardware assisted virtualization.

Xen is actually quite fast (far faster than BOCHs, or even normal Qemu... probably similar to KQemu), and has pretty good support in the Linux world (it was the Big Thing until KVM basically stole most of its thunder). IIRC, Xen's architecture relies on a privileged Operating System running in 'Domain 0' (aka dom0), generally Linux, which is given special permission to control the other virtual machines (dubbed 'domain U' or 'domU'), as well as direct access to the hardware. The domU machines wouldn't be given direct access to the hardware (normally at least- Xen at least used to support PCI pass-through, so, say, if you had a second network adaptor, you could basically connect it directly to one of the VMs). The memory could be allocated directly to the VM when it begins booting up (ala Qemu/Bochs). Another option would be to use memory hotplugging (if the OS supports it), or use a custom driver that upon boot up reserves most of the system's memory for itself (which it then tells the hypervisor it controls and the HV can take it back for itself), and as more memory is needed the driver can ask the hypervisor for chunks back, which it then 'frees' back to the virtual machine's operating system to use.


Although, architecturally, the OP's idea sounds a bit more like KVM (not to be confused with a KVM switch), as the virtual machine monitor seems like it would be integrated directly into the operating system, and not really a stand-alone piece of code that depends on a 'dom0' to control the devices.

Edit: I wrote 'VT-d' by mistake, VT-d is Intel's version of an IOMMU (to make it possible to virtualize DMA), while 'VT' is their virtualization extension which is really what made hardware assisted virtualization possible on x86 (partly, it added a "Ring -1").

[gravaera]

Oh, nah. For the first three points I made I was referring to a hardware supported setup, not virtual machines. As in, actually relinquishing control of the CPU itself to a client OS. That's why i mentioned the sysRq, and whatnot.

But yeah: I still do need to do some solid reading on virtual Machines. And I'm STILL not completely done with the Intel Manuals (only on vol 3A)...

[Brendan]

Hi,

Although, architecturally, the OP's idea sounds a bit more like KVM (not to be confused with a KVM switch), as the virtual machine monitor seems like it would be integrated directly into the operating system, and not really a stand-alone piece of code that depends on a 'dom0' to control the devices.

Yes.

The problem isn't whether or not it can be done, but how Ratty is planning to do it - there's lots of options (and lots of potential complications)...


Cheers,

Brendan

[Kitsune]

The problem isn't whether or not it can be done, but how Ratty is planning to do it - there's lots of options (and lots of potential complications)...

Most definitely! Although, IMO, that's also the fun part !

Fortunately (or unfortunately ), there's a good bit of writing on x86 virtualization now, including stuff like Xen And the Art of Virtualization (IIRC, that pre-dates the hardware assisted approaches and focuses on paravirtualization), as well as classic virtualization writings going back as far as the dawn of computing.

IMO, I'd say if your hardware supported it, focusing on using VT (+VT-d) would probably be the best approach, as you can depend on the hardware for a lot of the work... the downsides being that you basically cut out anyone with an older machine (which may include you!), as well as current (or at least the original) versions being a bit slower than optimized paravirtualization. Decisions like hardware-assisted or not are probably one of the first decisions Ratty should make, as that drastically changes the other options that are available to him.

[Brendan]

Hi,

Fortunately (or unfortunately ), there's a good bit of writing on x86 virtualization now, including stuff like Xen And the Art of Virtualization (IIRC, that pre-dates the hardware assisted approaches and focuses on paravirtualization), as well as classic virtualization writings going back as far as the dawn of computing.

IMO, I'd say if your hardware supported it, focusing on using VT (+VT-d) would probably be the best approach, as you can depend on the hardware for a lot of the work... the downsides being that you basically cut out anyone with an older machine (which may include you!), as well as current (or at least the original) versions being a bit slower than optimized paravirtualization. Decisions like hardware-assisted or not are probably one of the first decisions Ratty should make, as that drastically changes the other options that are available to him.

Or (for the more adventurous) have a generic framework with (load on demand) "engines", where one engine uses Intel's virtualization extensions, one engine uses AMD's virtualization extensions, one does dynamic translation, one does interpreting (for debugging purposes), etc.


Cheers,

Brendan

[Kitsune]

Or (for the more adventurous) have a generic framework with (load on demand) "engines", where one engine uses Intel's virtualization extensions, one engine uses AMD's virtualization extensions, one does dynamic translation, one does interpreting (for debugging purposes), etc.

I like how you think!

Such an architecture (even if only one engine is ever implemented) would also force you to spend more time thinking about how it should all work, hopefully resulting in a nicer design in the end. It'd be interesting if you were able to design it in a way where you could switch engines on the 'fly' (would be nice if you wanted to switch to the interpreter mid-session to debug something).

[Brendan]

Hi,

Such an architecture (even if only one engine is ever implemented) would also force you to spend more time thinking about how it should all work, hopefully resulting in a nicer design in the end. It'd be interesting if you were able to design it in a way where you could switch engines on the 'fly' (would be nice if you wanted to switch to the interpreter mid-session to debug something).

Some hyper-visors/emulators allow virtual machines to be suspended and migrated to other (host) computers (I even think there's one that will migrate virtual machines while they're running - having trouble remembering which one though). Switching engines would be similar (but probably much easier)...

Of course poor Ratty is probably starting to panic now...


Cheers,

Brendan

[Kitsune]

Some hyper-visors/emulators allow virtual machines to be suspended and migrated to other (host) computers (I even think there's one that will migrate virtual machines while they're running - having trouble remembering which one though). Switching engines would be similar (but probably much easier)...

Xen and KVM both support "live" migration from a quick google search... so I guess in addition to allowing dynamically changing the engines, Ratty should also include live migration!

Of course poor Ratty is probably starting to panic now...

And I'm loving every moment of it!

[Colonel Kernel]

The next version of Hyper-V will also support live migration.

@Ratty: There are some interesting white papers on the architecture of Hyper-V that might give you some design parameters for what you want to do.

[Ratty]

Panic? no

I was thinking of having 'engines' as suggested and that support for virtualisation would be an essential part of the operating system itself. From your responses I now know that it certainly is possible in theory so I shall be working, as a hobby, towards this goal and will keep you updated - more than likely through asking questions about hitting roadblocks. It will be a very long time before I have anything worth mentioning or even able to screenshot but some of you might be interested in the IDE I'm making so I will probably post about that at some point.

Thanks for all your replies, they were very informative.

[AndrewAPrice]

Do you mean something like VMware ESX? It's VMWare, running directly on the hardware without a need for a host operating system (it is the host).