Page 3 of 8

Re: Can you run your apps on each other's operating systems?

Posted: Fri Nov 05, 2010 2:27 am
by Solar
DavidCooper wrote:Bear in mind that my goal is to create artificial intelligence to the point where it can disassemble any program and work out exactly what it does in every aspect...
Simply impossible. See Halting Problem, and its consequence Rice's theorem. Again: It has been proven to be impossible.
My eventual plan is for A.I. to write all the device drivers...
Ah, I see where you're coming from. That's a problem I also had, years ago. Some exposition to the reality of software and hardware development will solve that over time...
How is code going to be injected into a browser from the outside? If I could connect a machine with my OS to the Web and wrote a browser for it, you could send it as much malicious code as you like and it would never be run.
Buffer overruns spring to mind, or SQL injections.

It doesn't have to be intentional malware to cause damage. Simple human error is completely sufficient.
No non-trivial system is bug-free.
Many bugs are harmless and can be left in, but harmful ones show up and can be eliminated.
That's my 9-to-5 job. And I can tell you two things: 1) Whether a bug is "harmless" or not cannot be told, as even trivial bugs can be exploited in new and interesting ways. And 2) even oh-my-god-how-could-anyone-have-missed-this bugs can remain undiscovered for ages.

Memory protection is easy. Hardware for MP is available in every desktop CPU today. Its performance impact is almost negligible on those same contemporary CPUs. And even if your "A.I." concept would be working (which it doesn't), that's no reason not to have a second layer of defense - in my book.
...I have no doubt that browsers can be designed in such a way that it would be impossible for them to run code they aren't meant to be running.
Might have been true once. I can tell you that HTML 5 requires your browser to "run code" that you have no control over. All you can do is to properly sandbox your browser process so that malicious code doesn't do damage.

Re: Can you run your apps on each other's operating systems?

Posted: Fri Nov 05, 2010 4:42 am
by gerryg400
I'm really with Solar here. Anyone who thinks he can design, implement, test and maintain a system that has 0 defects is delusional. Don't forget that the specs we follow, hardware we use, tools etc. all have bugs. Most popular software comes with warnings and disclaimers. There is a reason for that. There is no way a software company would ever guarantee that its software is bug or exploit free.

Look at this http://www.computerworld.com/s/article/ ... indows_bug and http://www.cve.mitre.org/cgi-bin/cvenam ... -2010-0232. Remember that this bug is in software used by hundreds of millions of users over a 17 year period. No in-house or beta testing ever found it.

Re: Can you run your apps on each other's operating systems?

Posted: Fri Nov 05, 2010 12:42 pm
by Solar
It's very easy to drive your car defensively and safely. Traffic is well-regulated, nothing should happen really. You even have to be registered before you can take to the streets.

Nevertheless engineers put safety belts and airbags in cars.

Re: Can you run your apps on each other's operating systems?

Posted: Fri Nov 05, 2010 6:07 pm
by DavidCooper
Solar wrote:
DavidCooper wrote:Bear in mind that my goal is to create artificial intelligence to the point where it can disassemble any program and work out exactly what it does in every aspect...
Simply impossible. See Halting Problem, and its consequence Rice's theorem. Again: It has been proven to be impossible.
I'll rephrase what I said before: ... to the point where it can dissassemble any program and work out exactly what it is indended to do in every aspect. Of course you can write programs that can run forever and do unpredictable things, but to try to write a virus in such a way that your intention can be kept hidden within such a program is another matter entirely. How deep can a programmer think when trying to hide complex malicious code within mathematical algorithms which never halt? How many programmers normally write apps that produce unpredictable data which is then run as code? The problematic programs that mathematicians are interested in have nothing to do with real-world apps.
How is code going to be injected into a browser from the outside? If I could connect a machine with my OS to the Web and wrote a browser for it, you could send it as much malicious code as you like and it would never be run.
Buffer overruns spring to mind, or SQL injections.
Is it possible for a buffer overrun to happen without you making an error in calculating how much space the data coming in will take up? I'd assumed that any device bringing in data without feeding it through the processor would have a counter which you set first to tell it when to stop. The DMA certainly does when I use it for disk input, but I've no idea what goes on with more complex modes and never-ending input via USB. I don't know about SQL injections either - do they just fire out machine code into random memory locations without you asking them to?
It doesn't have to be intentional malware to cause damage. Simple human error is completely sufficient.
And if the protection mechanisms catch the problems, why bother to remove the bugs? Just put the software out as it is and never mind if it's badly written. The app might occasionally have to be shut down, but it won't affect anything else running in the machine. I don't like that mentality, and nor did the people who wrote all the stuff that runs on my Psion 5mx which I've been using heavily for years without a reboot.
That's my 9-to-5 job. And I can tell you two things: 1) Whether a bug is "harmless" or not cannot be told, as even trivial bugs can be exploited in new and interesting ways. And 2) even oh-my-god-how-could-anyone-have-missed-this bugs can remain undiscovered for ages.
There are plenty of unimportant display bugs which can safely be left alone if they aren't too distracting (usually it requires extra code to clear them up), and no one can possibly exploit them to do harm on the machine.
Memory protection is easy. Hardware for MP is available in every desktop CPU today. Its performance impact is almost negligible on those same contemporary CPUs. And even if your "A.I." concept would be working (which it doesn't), that's no reason not to have a second layer of defense - in my book.
I just have a feeling that it may disappear from machines in the future, so why design an OS to depend upon it? I can easily add it to my OS and run certain programs in their own address space, and the whole point of this thread was to find out if there was a system out there which I should be basing it on in order to use a compatible API. But programs from trusted sources can also run in the same address space as the OS where they can be made a good bit simpler and a fraction faster.
...I have no doubt that browsers can be designed in such a way that it would be impossible for them to run code they aren't meant to be running.
Might have been true once. I can tell you that HTML 5 requires your browser to "run code" that you have no control over. All you can do is to properly sandbox your browser process so that malicious code doesn't do damage.
I haven't read up on HTML 5 yet. What is the code supposed to do, and what form does it take? I can't imagine that it would just inject machine code from any old site on the Net into your machine's memory and force you to run it.

Re: Can you run your apps on each other's operating systems?

Posted: Fri Nov 05, 2010 7:27 pm
by NickJohnson
DavidCooper wrote:I just have a feeling that it may disappear from machines in the future, so why design an OS to depend upon it?
Paging is not going anywhere. Intel chips still have backward compatibility for twenty year old code: features are clearly hard to repeal. Paging has very few shortcomings (unlike segmentation), and there are no better memory protection schemes at the moment, so we are at the very least going to have access to it for another twenty years.

Re: Can you run your apps on each other's operating systems?

Posted: Fri Nov 05, 2010 7:33 pm
by Casm
Clearly some of you have created an OS purely for fun or learning purposes and have no real expectation of running anything much on top of it, but quite a few of you appear to have dreams about making your OS a big player on the world stage some day. That is my hope for my OS too, but what chance will it have if it can only run applications written specifically for it?
Well I suppose nothing can last for ever, and that includes Microsoft/Windows. Nevertheless, hobbyists like us have next to no chance of knocking Microsoft off its pedestal. The only way your operating system might be commercially successful would be if you could identify a highly specialised niche, and fill it.

Re: Can you run your apps on each other's operating systems?

Posted: Fri Nov 05, 2010 10:30 pm
by KotuxGuy
berkus wrote:
KotuxGuy wrote:
berkus wrote: By alternatives I meant thinking outside the "browser/office package" box.

But every OS these days has a web browser.
Exactly, and browser is just a glorified overly complex shell to access the information you could receive via more efficient ways.
What ways? A web browser is the only way(save from using netcat, bet you've never used that) to access web sites.
Casm wrote:
Clearly some of you have created an OS purely for fun or learning purposes and have no real expectation of running anything much on top of it, but quite a few of you appear to have dreams about making your OS a big player on the world stage some day. That is my hope for my OS too, but what chance will it have if it can only run applications written specifically for it?
Well I suppose nothing can last for ever, and that includes Microsoft/Windows. Nevertheless, hobbyists like us have next to no chance of knocking Microsoft off its pedestal. The only way your operating system might be commercially successful would be if you could identify a highly specialised niche, and fill it.
The Linux kernel started as a hobby, and it's getting closer than any other OS to knocking off Windows(no offense to Mac or BSD users, I don't know too much about them), AFAIK.

And let's not forget the truly massive GNU project...

Re: Can you run your apps on each other's operating systems?

Posted: Sat Nov 06, 2010 2:25 am
by Solar
DavidCooper wrote:I'll rephrase what I said before: ... to the point where it can dissassemble any program and work out exactly what it is indended to do in every aspect.
Sigh. That sounds great at the discussion table. Have you ever tried to make sense of raw machine code? Yes, you can see that numbers are shoved here and there, that there are increments and decrements and conditional branches. But to figure out what that code actually does, on a semantic level?
Of course you can write programs that can run forever and do unpredictable things, but to try to write a virus in such a way that your intention can be kept hidden within such a program is another matter entirely. How deep can a programmer think when trying to hide complex malicious code within mathematical algorithms which never halt?
You haven't gotten the larger picture. What the Halting problem says is that there is no way a software could figure out what some other software does without actually running it with its actual parameters.
Is it possible for a buffer overrun to happen without you making an error in calculating how much space the data coming in will take up? I'd assumed that any device bringing in data without feeding it through the processor would have a counter which you set first to tell it when to stop.
In theory, yes. You need to forget about that only once. Or you restructure your code and the part that does the check suddenly disappears from the execution path. Such things happen.
I don't know about SQL injections either - do they just fire out machine code into random memory locations without you asking them to?
No, they execute arbitrary SQL commands because you forgot to sanitize your input. That could be a command to delete your database, to modify data or to introduce a subprogram.
It doesn't have to be intentional malware to cause damage. Simple human error is completely sufficient.
And if the protection mechanisms catch the problems, why bother to remove the bugs?
As I said, I know that argument. It was fielded for years by Amiga users to justify AmigaOS not having memory protection. Hell, I said so myself.

You know what? I was wrong.

There are good reasons to make your software as bug-free as possible. User satisfaction, for one. But there is no reason to take risks, to jeopardize your work of the last hour because the next time your mail program fetches mail it might croak and take the whole OS with it.
I don't like that mentality, and nor did the people who wrote all the stuff that runs on my Psion 5mx which I've been using heavily for years without a reboot.
My Amiga also ran long stretches without a reboot. Especially when it was running only a select few applications. But I still wanted to have memory protection. It's a bad feeling, not really knowing whether your work might have been corrupted by an application crashing (and even on the Amiga, where the quality bar for software was quite high and people took pride in their work, apps crashed occassionally).
There are plenty of unimportant display bugs which can safely be left alone if they aren't too distracting (usually it requires extra code to clear them up), and no one can possibly exploit them to do harm on the machine.
Says you. How many experience do you have with malware? A good friend of mine works in the security department of a major German bank. He could tell you some things about how "unimportant display bugs" and such stuff can be exploited, which would make your hair stand on end.
Memory protection is easy.
I just have a feeling that it may disappear from machines in the future, so why design an OS to depend upon it?
I bet you a month's salary against a hamburger that won't happen within our lifetime.
But programs from trusted sources can also run in the same address space as the OS where they can be made a good bit simpler and a fraction faster.
You said you want to keep track which memory belongs to which application, and which memory areas are accessed by which app. You can't do that significantly "simpler" than through the MMU.
I haven't read up on HTML 5 yet. What is the code supposed to do, and what form does it take? I can't imagine that it would just inject machine code from any old site on the Net into your machine's memory and force you to run it.
The friend I was talking about has found (and reported) a bug in the specification of HTML 5, that basically requires a compliant browser to execute code embedded in a canvas object. And that's just one miniscule thing in a very complex standard. (Just to prove that it's not only software errors. There are errors in hardware, there are errors in specifications. You cannot avoid that.)

Re: Can you run your apps on each other's operating systems?

Posted: Sat Nov 06, 2010 3:21 am
by rdos
Solar wrote:
Is it possible for a buffer overrun to happen without you making an error in calculating how much space the data coming in will take up? I'd assumed that any device bringing in data without feeding it through the processor would have a counter which you set first to tell it when to stop.
In theory, yes. You need to forget about that only once. Or you restructure your code and the part that does the check suddenly disappears from the execution path. Such things happen.
We also have initialized pointers, multithreading and such that could break things.
Solar wrote:
I don't know about SQL injections either - do they just fire out machine code into random memory locations without you asking them to?
No, they execute arbitrary SQL commands because you forgot to sanitize your input. That could be a command to delete your database, to modify data or to introduce a subprogram.
The problem is mostly with SQL itself, and sloppy implementations. Validation of input fields should not have been left to the application in the first place.
Solar wrote: As I said, I know that argument. It was fielded for years by Amiga users to justify AmigaOS not having memory protection. Hell, I said so myself.

You know what? I was wrong.

There are good reasons to make your software as bug-free as possible. User satisfaction, for one. But there is no reason to take risks, to jeopardize your work of the last hour because the next time your mail program fetches mail it might croak and take the whole OS with it.
Yes. Up until a few years ago I regularily validated our terminals in the most tightly protected environment possible. Every single malloc/new got its own selector, and it was run in protected mode. I found most of the bugs that I could never had found in the real-mode environment (without debugger) that the terminal run in. Now I have a new (page-based) algorithm that essentially does the same, which I enable from time to time to check for errors.
Solar wrote: There are errors in hardware, there are errors in specifications. You cannot avoid that.)
There is even reverse-enhgineered code made from real runs with the hardware in for examle Windows. Some of the device-drivers in Linux have been made in this way. Chances are such drivers won't handle exceptional conditions well, and nobody actually knows the quality of the drivers these companies put into Windows either.

Re: Can you run your apps on each other's operating systems?

Posted: Sat Nov 06, 2010 3:29 am
by rdos
NickJohnson wrote:
DavidCooper wrote:I just have a feeling that it may disappear from machines in the future, so why design an OS to depend upon it?
Paging is not going anywhere. Intel chips still have backward compatibility for twenty year old code: features are clearly hard to repeal. Paging has very few shortcomings (unlike segmentation), and there are no better memory protection schemes at the moment, so we are at the very least going to have access to it for another twenty years.
Agreed. Paging will not disappear anytime soon. It provides pretty good protection, but it could do much better (especially with a 64-bit address space). If a random 64-bit start address was generated for every malloc/new operation, chances of malicous or accidental corruption would be close to zero. Unfortunately, this would be very wasteful with the current paging scheme (it would usually need up to three physical pages per allocation).

Re: Can you run your apps on each other's operating systems?

Posted: Sat Nov 06, 2010 4:26 am
by Solar
rdos wrote:Up until a few years ago I regularily validated our terminals in the most tightly protected environment possible. Every single malloc/new got its own selector, and it was run in protected mode. I found most of the bugs that I could never had found in the real-mode environment (without debugger) that the terminal run in.
That reminds me: The most useful development tool on the Amiga, the one that allowed you to find the most errors in your code, was called "Enforcer". What it did, basically, was to retro-fit memory protection on those Amigas that had a MMU: Suddenly you saw when unallocated memory was accessed.

There's a lesson in there.

Re: Can you run your apps on each other's operating systems?

Posted: Sat Nov 06, 2010 5:04 am
by Casm
KotuxGuy wrote:
Casm wrote:
Clearly some of you have created an OS purely for fun or learning purposes and have no real expectation of running anything much on top of it, but quite a few of you appear to have dreams about making your OS a big player on the world stage some day. That is my hope for my OS too, but what chance will it have if it can only run applications written specifically for it?
Well I suppose nothing can last for ever, and that includes Microsoft/Windows. Nevertheless, hobbyists like us have next to no chance of knocking Microsoft off its pedestal. The only way your operating system might be commercially successful would be if you could identify a highly specialised niche, and fill it.
The Linux kernel started as a hobby, and it's getting closer than any other OS to knocking off Windows(no offense to Mac or BSD users, I don't know too much about them), AFAIK.

And let's not forget the truly massive GNU project...
Linux replacing Windows is a bit like the second coming. The true believers keep predicting it, but it never happens. The only company with enough muscle to do it if they really wanted to is Google, and even they would find it very difficult with Window's enormous installed base.

If a fundamentally new architecture comes along, offering something computer technology has ever been able to offer before, Microsoft ignores it as something here today and gone tomorrow, and then somebody comes along and produces a new operating system for it, then.....

Re: Can you run your apps on each other's operating systems?

Posted: Sat Nov 06, 2010 9:46 am
by KotuxGuy
Casm wrote: Linux replacing Windows is a bit like the second coming. The true believers keep predicting it, but it never happens. The only company with enough muscle to do it if they really wanted to is Google, and even they would find it very difficult with Window's enormous installed base.
I know it will take a while if Linux passes Windows, because traditionally, Linux has been seen as something for geeks, plus,
there's relatively few ways to get it installed on a PC without the user downloading an ISO, or buying a disc.
Casm wrote:If a fundamentally new architecture comes along, offering something computer technology has ever been able to offer before, Microsoft ignores it as something here today and gone tomorrow, and then somebody comes along and produces a new operating system for it, then.....
Like, say, Android? Which, incidentally, is backed by Google, and uses the Linux kernel(IIRC).

Re: Can you run your apps on each other's operating systems?

Posted: Sat Nov 06, 2010 11:07 am
by NickJohnson
Solar wrote:What the Halting problem says is that there is no way a software could figure out what some other software does without actually running it with its actual parameters.
Actually, no. It says that there is no way a program could figure out what every other program does. Because Turing-complete devices (a.k.a. humans and computers) write all existing software, Turing-complete devices can deconstruct all existing software semantically in finite time. This is why we as humans can usually understand how a program works by reading it. This is not to say that it's reasonable to try and do this. Whatever AI deconstructs the software would have to be extremely complicated, and probably would take exponential time relative to the time taken to write the software, assuming P!=NP. It's a bad idea, and will be for at least a couple hundred years into the future, even with advances in AI.

From Wikipedia:
While Turing's proof shows that there can be no general method or algorithm to determine whether algorithms halt, individual instances of that problem may very well be susceptible to attack. Given a specific algorithm, one can often show that it must halt for any input, and in fact computer scientists often do just that as part of a correctness proof. But each proof has to be developed specifically for the algorithm at hand; there is no mechanical, general way to determine whether algorithms on a Turing machine halt. However, there are some heuristics that can be used in an automated fashion to attempt to construct a proof, which succeed frequently on typical programs. This field of research is known as automated termination analysis.

Re: Can you run your apps on each other's operating systems?

Posted: Sat Nov 06, 2010 7:45 pm
by DavidCooper
OK, you've convinced me that there will be a role for memory protection for a long time to come and that it's well worth using it, but I still see no reason not to have an inner zone in the same address space as the OS where trusted apps can be run - those that can be analysed properly by A.I. (which would include any normal program that's well designed). I'm also quite happy to go on running my own apps in that zone because they don't contain bugs that bring down the system, though I'm also working on ideas in which apps work together as if they are a single app (or where most of their code is effectively a dll which other apps can build on just by linking in with near calls), so it would be silly to separate them. If I make an error in a program it's likely to crash the whole system, so that pushes me into getting things right first time - if you rely on your errors being picked up in less disruptive ways, you get into the habit of writing sloppy code which you haven't bothered to think through from every possible angle, and that opens the door to the kind of serious bugs that can go unnoticed for years. In addition, because I work directly with machine code, I can see all the possible things that might happen when a piece of code is run much more easily than someone who uses a programming language - everything happens as arithmetic and can be seen clearly in that form, but if you're working with abstractions, a lot of the picture is hidden from you and it's much harder to think through what the range of possibilities is.