A new distributed computing idea

[earlz]

Ok, I had this crazy but weird idea which seems mildly feasible. It would require an extremely different kernel from what we have out there right now though.

My idea: Rather than having this "live in the cloud" crap currently, why not "be the cloud"? Ok let's say we have an office of 20 computers and 2 servers. All of the computers use the same software and must access the 2 servers often.

Traditional approach to this would be either:
1. something like Novell
2. Install a regular OS on all of the computers and no applications and then have everyone Remote Desktop/VNC into the server to do work.
3. Keep all of the computers up to date with the same software

Of course, both of these approaches have problems. Novell still requires programs to be installed on individual computers. and doing Remote Desktop into the server increases server load a lot and wastes what computing power the clients have. On 3, of course, thats very high maintenance and could cost more money if there is licensing fees involved

My proposal: A cluster operating system that works with regular computers and can smartly balance load to any node. First off there is 1 Controller node. This really would have sorta 2 kernels. One for being like the rest of the nodes and another for managing all of the nodes.

It's difficult for me to explain....

What I intend on this cluster OS being able to do: being able to use a cluster of computers as 1 computer(with multiple users). So that if you logon to a 4 node cluster, and each node has 256M ram, 10G hdd, and basic P3 processors, you get a virtual 1G of ram, 20G hdd(wel, depends on configuration) and 4 virtual processors. It would be auto-load balancing. So say you logon to the cluster from computer A, and the rest of the computers are B, C, D.

First off, let's assume our configuration is that everything but /home is mirrored on all of these comptuers. So to access a non-/home file takes the same amount of time across all computers and for every writing of non-/home files, a message is broadcast to all computers to perform such a write.

All applications will have a preference of running off of computer A. But if you start using a lot of resources, then some computers may start on the most idle computer, B, or C or D. Well, it can send back screen drawing commands back to A, which is the physical computer you are using.

I can't figure out more to explain, so maybe if someone would just ask me some questions I could answer them to explain it.

Oh, and one more great thing. Because local mirroring can be done: when you take Computer A home with you, it has all of the applications as the server did, so you could force a `controller` mode where it can be stand alone and then you could use the computer without having the cluster. This is dependent on how things are mirrored and cached though.

sorry for the horrible explanation.. I'm having trouble just figuring out how to put my idea into words

[AJ]

Hi,

I like the idea and IIRC, this is the sort of thing that Brendan is trying to achieve.

One of the most difficult bits that I can see is responding nicely when one of the nodes loses connectivity and all or a part of its physical memory is used by other nodes. By comparison, load balancing or adding a new node is easy. IMO (not having studied distributed computing in a major way), you will end up with one of two things:
a) Instability. As we are talking about businesses here, this is not acceptable.
b) Huge redundancy. Ok if you have specced out the machines quite well, but if that's the case, why not just avoid the complexity and use the traditional local apps / file server model.

The OS also needs to be designed so that it works well in "standalone mode" which may be tricky to implement if it's too heavily biased towards the distributed idea. Perhaps design a decent OS and add the distributed layer on top?

Don't get me wrong, though - as I stated, I like the actual idea. There are just a few challenges to overcome first. Good luck if you go ahead with it!

Cheers,
Adam

[NickJohnson]

I actually had an idea like this for my OS originally, but even on a design level, there are a lot of things to overcome. The heart of most of the problems is that you can't trust other computers to do things correctly. If you export an important process to another node, but that node is broken in some way, the entire system could go down. Additionally, other nodes may not be trusted, so you can't always export processes that have sensitive information, like passwords. You also can't completely trust that other nodes are properly synchronized with general information about the system, like how many nodes there are, and where processes are that you need to communicate with, so there has to be some sort of centralized control, which ruins scalability.

I haven't really solved any of this, but I'm pretty sure this sort of thing has been done to different degrees elsewhere. I think this was one of the goals of Amoeba, a relative of MINIX.

[Brendan]

Hi,

I like the idea and IIRC, this is the sort of thing that Brendan is trying to achieve.

Yes.

People (mostly monolithic kernel advocates) often complain about the extra latency due to IPC in micro-kernels on a single computer (because of the overhead of task switches, TLB flushes, etc). When you add the latency of networking to this it becomes much more significant; and one of the most common techniques used to improve IPC performance (shared memory) fails badly on distributed systems (it takes a huge amount of overhead and latency to keep areas of RAM on separate computers synchronised).

The only sane way to solve the IPC latency problem is to send a request and do other work while you're waiting for a reply. For synchronous IPC this means spawning lots of threads (one per request, which means more hassle/overhead for the scheduler, and only really pushes the problems elsewhere). The other alternative is asynchronous IPC - for example, a single thread could/should be able to send 50 requests (to local or remote receivers), then do other work while it waits for 50 replies.

Asynchronous IPC does make it difficult to write complex software, because you end up with a central message handler that needs to track the state of many different things at the same time. To solve that you need to split things up into many smaller/separate pieces that are easier to manage. For example, rather than writing a text editor as one piece of sequential code, you'd split it into maybe 3 different pieces - one piece to handle interaction between the application and the user (keyboard, mouse, video), one piece to handle the document itself (e.g. loading/saving, getting lines of text from the document, inserting/deleting text in the document, etc), and a third piece that ties it all together (e.g. 3 objects, with a central message handler for each object). The interesting thing is that the "asynchronous messaging with one object per thread" idea scales very well (even on a single multi-CPU computer), and can be done without the need for re-entrancy locking (and problems like contention, live-lock, dead-lock, race conditions, etc); which can make it an attractive idea for non-distributed systems too.

However, for asynchronous IPC you'd need to forget about POSIX and forget about porting any existing software to your OS. For most hobby OSs this is a massive problem - it's a lot easier if you can write a kernel, then port GCC, make, BASH, etc to your kernel. If you can't port software to your OS, then you could spend 5 years writing your own compiler, etc.

I haven't really solved any of this, but I'm pretty sure this sort of thing has been done to different degrees elsewhere. I think this was one of the goals of Amoeba, a relative of MINIX.

Amoeba (and most existing distributed systems) use the "client/server" model. For example, you might have 5 computers running as X clients that don't do any general processing or storage, another 5 computers being used as "application servers" that do all the processing, one computer to keep track of the file system's directory structure, and 4 computers used for storing blocks of data for the file system. Mostly it's a reliability nightmare as there's very little redundancy - with 15 computers there's 15 times as much chance of one computer crashing, and 15 times as much chance of losing functionality.

What I'm working on is "peer-to-peer distributed", where all computers do all things and no central point of control (or, no single point of failure). For example, you could have 10 computers on a LAN in Australia and another 8 computers on another LAN in America, where both LANs are connected by the internet. If the internet connection goes down then it behaves like 2 separate clusters until the network comes back up again, and then the OS seamlessly resynchronises to become a single cluster again, without any loss of functionality (and without any of the users knowing there was a problem).

I've already solved the technical/design difficulties (including getting it to work with a mixture of different architectures in the same cluster). My problem is getting a high quality implementation done (I worry that if I release a poor quality implementation then it'll be dead before anyone bothers taking a closer look).


Cheers,

Brendan

[quok]

Ok, I had this crazy but weird idea which seems mildly feasible. It would require an extremely different kernel from what we have out there right now though.

My idea: Rather than having this "live in the cloud" crap currently, why not "be the cloud"? Ok let's say we have an office of 20 computers and 2 servers. All of the computers use the same software and must access the 2 servers often.

[snip]
My proposal: A cluster operating system that works with regular computers and can smartly balance load to any node. First off there is 1 Controller node. This really would have sorta 2 kernels. One for being like the rest of the nodes and another for managing all of the nodes.

It's difficult for me to explain....

What I intend on this cluster OS being able to do: being able to use a cluster of computers as 1 computer(with multiple users). So that if you logon to a 4 node cluster, and each node has 256M ram, 10G hdd, and basic P3 processors, you get a virtual 1G of ram, 20G hdd(wel, depends on configuration) and 4 virtual processors. It would be auto-load balancing. So say you logon to the cluster from computer A, and the rest of the computers are B, C, D.

I'm wondering how this is any different from using a SSI (single system image) capable OS/kernel along with a caching distributed filesystem. Think openMosix (a defunct linux kernel extension that implemented SSI) along with the Andrew File System.

Sorry, but this idea of yours isn't anything terribly new. I used both openMosix and AFS 5+ years ago to do exactly what you are talking about here.

[earlz]

My idea is different from SSI because remote computers can use IPC and such across network.

Yes, the reliability bit is a problem with my design. I'm not sure how to allow computers to delegate tasks to other computers without either a very different way of writing programs, or without reliability problems. Because if Process A is running on Computer B and it goes down, then Computer A no longer has access to it, so it "crashed" from the users point of view. Redundancy on the CPU level seems very ridiculous as well. So I'm stuck on this design now because that bit must be fixed.

[scgtrp]

My idea is different from SSI because remote computers can use IPC and such across network.

Again, already been done.

Because if Process A is running on Computer B and it goes down, then Computer A no longer has access to it, so it "crashed" from the users point of view. Redundancy on the CPU level seems very ridiculous as well. So I'm stuck on this design now because that bit must be fixed.

Disregarding hardware failure and differences between CPUs, if a process crashes on computer A, given exactly the same input and environment it'll crash on computer B as well. Even heisenbugs have a cause somewhere in code and are reproducible if absolutely nothing changes.*

*Except race conditions, of course, but any program relying on the result of a race condition deserves to crash.

[Owen]

*Except race conditions, of course, but any program relying on the result of a race condition deserves to crash.

Like Microsoft COM? Seriously, theres a well documented race condition in the way in which it unloads libraries (They work around this by instituting a 1 minute delay)

Though, to be honest, unloading libraries is Really Hard(TM)

[earlz]

My idea is different from SSI because remote computers can use IPC and such across network.

Again, already been done.

Because if Process A is running on Computer B and it goes down, then Computer A no longer has access to it, so it "crashed" from the users point of view. Redundancy on the CPU level seems very ridiculous as well. So I'm stuck on this design now because that bit must be fixed.

Disregarding hardware failure and differences between CPUs, if a process crashes on computer A, given exactly the same input and environment it'll crash on computer B as well. Even heisenbugs have a cause somewhere in code and are reproducible if absolutely nothing changes.*

*Except race conditions, of course, but any program relying on the result of a race condition deserves to crash.

Well I'm meaning if Computer B loses network or power connectivity.

And I mean that running the same process twice for each computer(as redundancy) is just mad.

[iammisc]

If you really want to pursue this, then you should join my open-source project, meerkat (http://meerkat-dist.sourceforge.net). We share pretty much the same goals: running many applications across multiple computers.

[whowhatwhere]

The concept of a distributed hash table migh be interesting here, though it's already done I believe.

[cxzuk]

@earlz

Your original post really tries to cover multiple areas of computing. The "cloud" is a combination of all of them =]

Firstly. Data. Many people already use a dedicated server or NAS to store data. An application is no different. Having the binaries located across a network is a simple problem to solve. NFS and UNIONFS could be used in linux as a crude implementation. Tho I believe the SSI is the same thing? - Alternatively just set up your own APT repo?

The main point about cloud computing is nothing to do with applications. Its the fact that your data follows you. You can access your data from any (permissions granted) available computer. (think hotmail or gmail).

Secondly, a Process.

What your talking about here is symmetric multiprocessing (SMP) and asymmetric multiprocessing (aSMP). SMP is when the load is balanced equally over all processors. aSMP is when you have a controlling node which tells each processor what to do.

SMP requires shared memory, which is a problem when latency is high. So as you concluded, a controlling node is needed in a aSMP style system.

If you want do really make SMP, you have to treat applications differently. Firstly, the state of the application must be kept by the client. (This is a rule from REST, which is related to web applications which are actually technically the same to aSMP).

In an IPC environment, this shouldn't be an issue, basically it means that any function must have all its parameters given to it, and any changes must be returned by the function and processed by the client. e.g.

var x = 0

function add_1_to_x() {
x=x+1
}

is not possible in a aSMP system. This has to be run locally. What you really want is;

var x = 0

function add_1(y) {
return y + 1
}

x = add_1(x)

the function "add_1" can then be executed by any connected node. Fall over and errors do not become a problem. Just retry the request =].

The concept of trust is exactly the same as current networks. SSL/TSL and certificates can be used to Authorise and Authenticate requests and replies. You would also build the node list on the client, which contained trusted servers. And not do a full broadcast (unless part of a VPN?).

MikeyB

[iammisc]

OP, I know exactly what you're thinking. It's what my open-source project meerkat (http://meerkat-dist.sf.net) aims to do. We've already written a networking infrastructure and distributed object system and we have a working execution system. In fact, with the addition of lists as a data type, the entire system will be turing complete.

I encourage all of you who have an interest in this field, not to reinvent the wheel, but to join a project who has already done all this. E-mail me at [email protected].