Software memory protection

[BrightLight]

I was wondering how to prevent unauthorised memory access in software, rather than hardware (without paging/segmentation).
Let's say I have two programs running and a memory heap at 1 MB. Program A allocates some memory, and program B allocates memory after it. Let's say program A allocated memory at 1 MB and program B did it at 1.5 MB. What prevents program A from writing more than 0.5 MB and corrupting program B's memory? How do I detect this, recover from this and prevent this?
I want theory and explanation, not code.

[SpyderTL]

I believe the short answer is, you can't.

This is why hardware memory protection is pretty much mandatory if you want a stable system.

The only real software option would be to run your applications under a virtual machine, were you used software to interpret the instructions, and emulated the computer hardware. But performance would probably be an issue.

I think hardware memory protection is the only real option, even though I'm not currently using it in my system.

I am primarily using stream readers and writer objects to read/write from/to memory, storage devices, and network connections. Otherwise, I just make sure that every part of the system gets its own "designated" memory area.

[alexfru]

In general, you can't cleanly recover once B corrupted A's memory. You'd need to know the lost values for you to reinstate them before A reads them again. And then B now has its own expectations for the corrupted locations as it wrongly believes they belong to it and should have B's values. So, in order for both A and B to keep running and using the same memory locations, you'd need some sort of copy-on-write mechanism, substituting A's physical memory (or values) when A reads it and B's physical memory (or values) when B reads it. And, of course, without special hardware or expensive software solutions you can't have that.

Without hardware support, your only options are not using dynamically allocated memory (and pointer arithmetic) or inserting the necessary run-time checks into the code to guarantee that A uses A's memory and B uses B's memory (and, perhaps, some additional checks as well since A or B may be able to corrupt itself). And like I said above, you still generally can't recover the corruptor.

[Rusky]

If you enforce that all applications are written in a memory-safe language, you can do this without hardware support or an interpreter. You'll need some run-time checks (mostly for array bounds) but nowhere near the overhead of an interpreter.

[Combuster]

Software protection requires a completely different perspective. Instead of running code and detecting when it misbehaves, you only run code that can not misbehave in the first place. In addition, even hardware protection actually prevents corruption of other processes, so in neither case there is any form of recovery. The kind of recovery you're looking at is the kind you'll see in spacecraft, where radiation from outside the system screws you up, unlike the software inside.


Software protection of traditional languages requires emulation or at least heavy instrumentation so that all the options once available in that language will actually still work. Several modern languages are specifically designed not to allow arbitrary memory access, hiding them behind usage patterns. Because the compiler generates the memory accesses and know in which context they take place, the compiler knows exactly which checks need to applied where, resulting in far more efficient code.

All of this has already been done in the past.