OSDev.org

Ok this is several thousand steps ahead of where I am currently with my OS but how practical would this be as an idea and what do you think it would take to make it work...

With the proliferation of laptops and sub-laptop like devices (origami, Nokia 770 etc) how about the ability to shift entire active running programs across networks? So that you could be working on a document in a Word processor and decide to shift it across to a tablet PC to take to work with you but leaving the entire thing in its current state. Is this possible? Feasable? Sensible?

I think there would not be enough benefits for this. There are only few times you work on something and at that moment have to switch devices. Also it would be very hard to implement because one executable has to work on multiple devices. In that case you'ld either have to write multiple oses or find others to make their oses compatable with yours.

Actually, I think the benefits would be enormous. Just look at the movie Minority Report for some ideas on how that could work.

Personally I think it would be great if I'm working on something or perhaps reading a document I can just shift it over to a small mobile computer when I need to rush out the door. I can then continue it when I board the train to work. Or better yet my self-driving car.

In that case I could perhaps also send my "state" over to my car in the first place.

I'd say endless possabilities, however very hard to do. Especially if you want to transfer state of something that is connected to something else.

For example. If I'm playing mp3's from a playlist or listen to the radio I cannot (succesfully) transfer that state to a remote machine that has no access to the list of mp3's or the radio signal.

Now in some cases you might be able to transfer the collateral state as well, but that can take a lot of time. I guess this becomes easier when everything is connected to everything in, say, 10 years (or so?).

Interesting concept!

Each device may have a different architecture/platform/processor or whatever you call it, so transferring a running program without ysing some sort of emulation won't work very often.

If you cleanly separate the document data from the rest of your program, it should be easy to transfer the document and info about current position in the document or whatever else is relevent, without transferring the complete state of the program.

HTH,
Paul Barker

Surely tho an operating system with suitably abstracted hardware layer will present the same underlying interface to the program on any machine... if not then there is always the possibility of running such things in a VM environment such as Java or MSIL where the platform remains identical by defination.

Hi,

elaverick wrote:With the proliferation of laptops and sub-laptop like devices (origami, Nokia 770 etc) how about the ability to shift entire active running programs across networks? So that you could be working on a document in a Word processor and decide to shift it across to a tablet PC to take to work with you but leaving the entire thing in its current state. Is this possible? Feasable? Sensible?

There are distributed OSs that can do "process migration" already, but for most things it normally doesn't work. For a quick overview of some of the problems, have a look at this web page.

IMHO there's 4 reasons why most distributed OSs have problems with process migration:

• * The designers build the distributed OS as an abstraction layer on top of an existing non-distributed OS, or begin with a non-distributed OS and try to add distributed features. This saves a lot of work, but leads to an inefficient mess that only half works.
  * The developers use a standard "client-server" approach, which means all computers end up relying on the server. An example of this would be Amoeba, where there's "CPU servers", "file data servers" and "file directory servers" (where a workstation is like a dumb terminal running X, and is a client to these servers). All computers that are running any kind of server must be available to the clients at all times.
  * They try to make it compatible with POSIX, which was never designed to handle this sort of thing.
  * It isn't easy to start with.

I believe it's entirely possible to do a seamless peer-to-peer distributed OS, where processes can be migrated to any other computer while they're running (even migrating process between entirely different hardware - like from 80x86 to/from Itanium or Sparc). I also think it's possible to allow computers to be turned off or disconnected with little impact on running processes. IMHO to do it properly the entire OS and all software that runs on it (and all APIs) must be designed for it.

This is what my project is all about. Unfortunately, I'm still getting the foundation done (boot code, micro-kernel, basic device drivers, etc), and I'm trying to do things "properly".

I also think the benefits are huge (resource balancing, fault tolerance, reduced hardware costs, easier administration, greater flexibility for users, etc). I won't give full details - it'd sound too much like a marketting brochure...


Cheers,

Brendan

Brendan wrote: I believe it's entirely possible to do a seamless peer-to-peer distributed OS, where processes can be migrated to any other computer while they're running (even migrating process between entirely different hardware - like from 80x86 to/from Itanium or Sparc).

I'm curious about how do you actually plan to accomplish this - migrating running processes to different machine architectures? I mean, it's quite obvious that you can't just copy the process image to the destination machine. Different machine code, different data structures (because of wordsize and endianess), ok.

So I see two possibilities:

• Integrate process migration into the application model: There must be a way to tell a running process to export all its state in a machine independent way, and to restart the process with this state on a different machine (with the side effect that you can suspend any process to disk ) But that would mean that a lot of responsibility lies in the hands of the application programmer, and if he/she (for some reason) implements the migration logic in a buggy way, the feature will be missing until it is fixed...
• Run processes in a VM, launch them from intermediate code, let the OS have as much insight into internals of the application as possible (schemas for data structures, for example). This way the OS could convert a process into a machine independent image without help of the application, and another machine could import and continue the process as well.

Which way did you choose (or is there even another way)?

cheers Joe

As I see it the only (real?) way to do that "properly" is to use a language like Java or .NET that *can* be platform independent. In my opinion you will also need support from the program for the transfer to succeed. To make that as easy as possible you'll need to build a framework that programs interact with.

Hi,

JoeKayzA wrote:Which way did you choose (or is there even another way)?

You're quite astute...

The basic idea is that the OS tells the process to save it's state. The process minimizes it's state (which involves closing any non-essentail file handles, shutting down any non-essential IPC connections, etc) and then saves it's state to a file. After this the OS re-starts the process again and tells it to load it's state from the previously saved file.

This means the OS can re-start the process at a later date, or restart the process on a different computer (migrate the process). It also means you can do "live updates" - if a new version of a process becomes available and the user is using the old version, you can tell the old version to save state and then restart the new version.

The next step is to allow the new process to be started in a "wait for state" mode, so that it can be loaded into memory and do it's initialization before the state is ready. This minimizes the delay between "save state" and "restore state", which is important. If it's done fast enough it can be done without the user noticing.

Lastly, if the process has an 80x86 version, a Sparc version and an Itanium version, then all versions must use the same format for "state files" (including "endian-ness").

Of course it's not this simple. Often, some files must remain open and some IPC connections can't be shut down. In this case re-starting the process at a later date is impossible and the OS/kernel needs to transfer file handles to the other computer and do IPC redirection. If a process saves a file name as part of it's state and is then migrated, that file name must refer to the same file on the other computer. This means that the entire file system structure must be the same for all computers and the OS may need to copy file data from one computer to another during the migration.

Then there's things most OSs support like shared memory (I don't allow shared memory in any way) and device driver access (I use a micro-kernel and let the IPC redirection handle this).

For my OS this will all be optional. A process can support none of it (required for device drivers), support migration only (if kernel assistance is needed for file handles or IPC redirection), or support migration and suspend (if kernel assistance isn't needed).

[continued]

[continued]

Now, imagine a LAN consisting of 6 single-CPU Pentium III's and an emulator (like Qemu, but designed for BCOS). You could tell the emulator to emulate a NUMA machine with 4 domains and then run another OS on it. Each emulated NUMA domain would be a seperate process, so that a quarter of the emulated computer could be migrated by itself. You could replace the computer's motherboards one at a time and the "guest OS" would have 100% uptime, or pehaps add a seventh computer to the LAN to increase available resources.

Of course this isn't all I'm planning. Another idea is redundancy or fault tolerance (where multiple copies of a process are run on different computers at the same time). Any message sent to the "set of processes" is duplicated and sent to each process in the set, and any message sent from one process must match messages sent from it's clones. This way errors can be detected (and if 3 or more copies of the process are running, the errors can be fixed). If there's a hardware failure (one of the computers catches fire and blows up) the other copies of the process can be told to save state and you can restart a new "set of processes" without data loss. This'd be perfect for banks or something, where a minor hardware glitch could cost them millions.

Then there's checkpointing. Every 5 minutes tell the process to save state, make backups of any files and then restart it. This would allow you to restore the process from any point in the past.

It's a lot of work, it's hard to get working right and some things need to be sacrificed (performance mainly), but in the end it should be an extremely powerful system.


Cheers,

Brendan

And I thought I was planning it unthinkably big and never to be completed because it's too radical...

Hi,

Candy wrote:And I thought I was planning it unthinkably big and never to be completed because it's too radical...

It's my "magnum opus", or life's work...

I'm estimating it'll take me around 2 years to get the basic stuff out the way (boot code, kernel, device management, etc), and probably 4 years before it'll run basic applications on "generic" hardware...

After that I'm hoping others will help - GUIs, device drivers, applications, etc.


Cheers,

Brendan

Brendan wrote: It's my "magnum opus", or life's work...

I'm estimating it'll take me around 2 years to get the basic stuff out the way (boot code, kernel, device management, etc), and probably 4 years before it'll run basic applications on "generic" hardware...

After that I'm hoping others will help - GUIs, device drivers, applications, etc.

Mine should also be a life's work, but I'm more of the decent basis stuff... I'm working on (simultaneously) a few compilers (c, c++, assembler to go with it, interface languages for own idea and possibly some more), compiler writing tools (to make beforesaid quicker and easier), my kernel, drivers, userland libraries, an slight adaptment of the stl, a filesystem for removable media, a filesystem for fixed media, a user interface (only in my mind atm, just started to try out an idea on graphical font rendering without bitmaps) and a few programs to run in it (text editor, internet browser). My one idiocy is that I'm refusing any and all existing code and designs unless I fully agree with them, or for compatibility. ELF has passed my judgement (as first standard I actually endorse), FAT/NTFS/HPFS/BeFS etc. have all failed, so I'm making a new one, including linux driver etc....

Add to that a little touch of perfectionism (if I find a design flaw all code is chucked and rewritten to conform to the new design) and it's taking forever already . Positive points: I've gone from two studies to only a single full-time-plus-overwork-job so I've got loads of time on my hands, so I can catch up with 2 years of OS work. Have got some 20-30 hours a week now, so I'm starting to pick up pace again (with some non-direct-kernel projects first).


Brendan, it sounds like we'll be colleagues for a lifetime to come

Brendan wrote: Hi,

JoeKayzA wrote:Which way did you choose (or is there even another way)?

You're quite astute...

Well, ok, so far it looks like 'option #1' from my list. This wasn't meant in a bad way, I just couldn't think a solution that doesn't fall into one of these 2 categories.

@Candy: About perfectionism and refusing existing designs - this sounds quite familiar to me . I haven't yet completely thrown away the idea of supporting legacy APIs however, but just in form of cleanly seperated and absolutely optional compatibility subsystems. But for now, I doubt that my project will ever reach the point where such a subsystem will be required.

cheers Joe

JoeKayzA wrote: @Candy: About perfectionism and refusing existing designs - this sounds quite familiar to me . I haven't yet completely thrown away the idea of supporting legacy APIs however, but just in form of cleanly seperated and absolutely optional compatibility subsystems. But for now, I doubt that my project will ever reach the point where such a subsystem will be required.

You're not making a new compiler since you don't like the idea of not being able to port gcc... *feels like an idiot, but one I like being*


the legacy api thingy is a point of twist in my head... If you support them, they'll be used VERY extensively and you won't get anything for the new api, dooming it. If you don't support it you might get slightly more for your own api but you'll miss a load of users because you can't support the other. My current thought is to make it a pure-emulation-thing that isn't implemented for speed, so it'll be very slow at most times. I also hope my idea does bring the advantages I'm hoping for, which should make creating programs in my subsystem very attractive comparatively... *fingers crossed* of the 20 people or so I've told it irl only 2 thought it might be a good idea and only 1 understood the implications (and asked whether it was even possible).