Brendan wrote:For an OS that uses hardware isolation there's little overhead for CPU-intensive workloads (a little after a task switch, and a little for any kernel API calls and exceptions). For an OS that uses software isolation there's still array bounds tests, pointer checking, etc - basically all of the overhead is still there. I'd expect that hardware isolation would give better performance than software isolation for CPU-intensive workloads, especially if there's only one "ready to run" thread per CPU or there's some other reason why very little preemption occurs.
Your expectation matches the experimental results from section 4.4 of "Deconstructing Process Isolation":
The run-time overhead of language safety is slightly higher than hardware isolation for the Bartok benchmark
The Bartok benchmark is very CPU-intensive.
This tradeoff exists today even in conventional OSes. If you write code in a managed language and do a lot of array accesses in a tight loop, you're going to get lousy performance. One of the major advances in programming languages that will deal with this problem is
dependent types, which allow for invariants that relate types to run-time values (e.g. -- instead of int[] as an array of ints of unknown size, you could have a type like int[](length==n) where n is a run-time parameter, variable, or expression). Forbidding dynamic loading makes this approach much more tractable since all the code is available to the compiler to do inter-procedural optimizations.
Brendan wrote:If hardware isolation does give better performance for CPU-intensive workloads and software isolation gives better performance for I/O intensive workloads, then which method gives better performance for typical/common workloads that are a mixture of I/O intensive and CPU intensive? At which point do you reach the "break even" point, where both methods give equal performance?
The nice thing about building a system that supports both isolation mechanisms is that you could leave it up to the application developers to decide based on their own benchmarks.
Brendan wrote:The basic question here is "when is software isolation better?". IMHO this question is the question all sane OS developers would ask, and this question is the question that researchers working with software isolation would be expected to answer.
It's not up to the researchers to answer this question, because it is an engineering question, not a research question. Ask a researcher what's possible; ask an engineer what's best for a given set of circumstances.
Brendan wrote:But it's not an apples-to-apples comparison. An apples-to-apples comparison would be comparing the performance of software isolation on an OS designed for software isolation to the performance of hardware isolation on an OS designed for hardware isolation.
A good experiment varies as few variables as possible at a time. Comparing two radically different OSes introduces too many variables to reach a meaningful conclusion. I think you're still missing the point of the experiment...
Brendan wrote:If someone added support for software isolation to an OS that was designed for hardware isolation (like Linux) and then claimed that software isolation sucks completely (because of the hacks, etc involved in making it work), then would you be convinced that software isolation sucks completely?
No, but no one is claiming that hardware isolation sucks completely either.
Brendan wrote:Once the costs are understood, individual systems can choose to use hardware isolation when its benefits outweigh the costs, it's just a pity that there's so little information to help anyone understand these costs and make an informed descision.
That's because it's early days. This HIP/SIP comparison was only done a little under three years ago.
Brendan wrote:Colonel Kernel wrote:It's not worthless at all, because this is the first time (to my knowledge) such a direct comparison has been done. It is helping decision-makers figure out what engineering trade-offs to make. Sure, the comparison scenario is not perfect, but neither is Singularity... It is a research OS, a prototype. The only way to get the comparison you want is to build a complete, commercial-grade system that can function either way (hw isolation or sw isolation, both fully optimized). That's too expensive just to satisfy your curiosity!
What they've done is provided benchmarks that provide little useful information (or even misleading information) to help decision-makers make the wrong descisions.
So... engineers might make the wrong choice and waste some time and money following a blind alley? Do you think any OS dev organization worth its salt will jump head-first into something like this without doing some experiments of their own first? I think this situation looks a lot different to independent OS developers with small-to-zero budgets, versus Microsoft and its annual R&D budget.
Guess who benefits most from MSR...
Brendan wrote:They don't need to write 2 full-fledged OSs. How hard would it be to take an OS like Linux or Windows and disable all the things where paging has been used to improve performance or reduce RAM usage? If they did this and found that these optimizations reduce application startup time by 50%, make already started processes run 5% faster and makes the average process use 15% less RAM, then they'd be able to mention this to give their own "unsafe code tax" statistics some perspective.
This would be interesting, but it wouldn't mean much in terms of a comparison to software isolation. There are just too many variables, like the effect of "tree shaking" on RAM usage and app startup time, for example.
Brendan wrote:How hard would it be for them to provide benchmarks for a CPU intensive workload?
They did, it's just in a different paper (see above).
Brendan wrote:How hard would it be to run their WebFiles benchmark on another OS (Windows, Linux) and compare the performance to the performance they get running the benchmark on Singularity?
I don't know, because I'm not sure whether WebFiles was developed exclusively for Singularity or not. It's a question of cost/benefit, and like I said, the intent was to compare different configurations of Singularity, not to compare Singularity to other OSes. Singularity is not a commerical OS, so this would be largely pointless.
Brendan wrote:So, let me ask you the most important (IMHO) questions that the Singularity research should have already answered. Does software isolation provide better performance than hardware isolation in all cases? If the answer is "no", then in which cases does software isolation provide better performance? Why?
The answer is "no", it is already in one of the papers (see above), and I think in general the answer I would give is that SIPs are faster in situations where processes communicate a lot with each other, like for I/O.
Let me give you another example that demonstrates why this research and the HIP/SIP choice is about much more than performance. Consider this comparison: A web browser written in an unsafe language loading plug-ins as dynamic libraries, versus a managed browser that loads its plug-ins in SIPs in the same domain (address space) and communicates with them via zero-copy IPC. That would make for a very interesting experiment. If it's possible to achieve a comparable level of performance with the SIP-based approach (not even necessarily as fast, just close enough to be barely noticeable by end users), then the security benefits of going with separate processes IMO far outweigh the performance advantage of using unsafe code.
I think it's better to not have to pay for what you might not use (e.g. -- OSes that use software isolated processes).
