OSDev.org

Hi there
I was wandering is it possible to encrypt an OS and prevent people from accessing code. If the OS is written in assembly what stops another user from simply putting the OS through a disassembler and gaining access to the Kernel code? Is it possible to use some kind of encryption method?
I just cant get my head around how that would work since most things that use some kind of encryption use the OS to access hardware etc. How would the OS be able to input commands the the hardware would understand whilst also being encrypted, surly it would need the CPU to be able to decrypt itself?

Many thanks

Before proceed reading, there are critical rule / assumption:

1. If somebody has physical access to the machine, it's almost over - eg. one may simply plug the hard-disk into other computer and decode it.
2. The OS is not install on emulator, machine with hardware debugger or ICE - You have control on the distribution.
3. The OS itself may not be encrypted when executing, that's why (2).

Now you may think of how to make cracking it harder (to some extend, the level of security is depend on economy, ie. to make it not worth cracking if the effort is high), I can think of some idea:

1. memory protection.
2. block access to critical files.
3. apply encryption to files increases the difficulty and hence increase security.

The point is isolation, the concept is similar to you can't access something not exposed on a web server.

Thanks for the fast reply.
So in the case of an OS similar to windows were it would be directly installed on the machine this would be near on impossible to actually secure. Thats what I thought. In theory could you have an OS that did the hardware and acted as a main Kernel. Then secondary to this have another OS running in a virtual machine? Could the secondary OS then be encrypted using the virtual machine?

Is this getting to a stage of theoretically you could, however it would take far to long to do, and would still be possible to hack if a groups of people wanted to such as lulzsec, annonymous, Rarzr1911 etc.

Obviously for my own little OS that I've started working on it wouldn't be worth having it if I gave it for free let alone hacking it .
Its mainly a theoretical question just to satisfy my curiosity.

eskimo456 wrote:So in the case of an OS similar to windows were it would be directly installed on the machine this would be near on impossible to actually secure.

We have to disagree on the definition of secure. In my opinion the access policy of OS code, the encryption model and the logic has little to do with security of user data, which is the one and only one thing matter.

However, security of user data is a huge field and way beyond my expertise.

As far as modern day security, there are two aspects to it. For commercial use, you shouldn't allow people breaking in from the outside without any intervention. That risk is quite manageable by keeping software up to date and having audits for your own work.

The greatest threat is actually the user breaking the security on behalf of the malicious party. Most of today's miseries are caused by phishing and social engineering rather than actual exploits.

In my previous job we used full disk encryption on all portable PCs and hard drives. Specifically we used a product called PointSec. I don't know much about it, I was just a user, not an administrator. It secured, to some extent, all the files on your laptop.

http://www.checkpoint.com/pointsec/

I think I may have used the wrong terminology and given the wrong impression. I am more curious in protection in terms of an intellectual property/copyright perspective as opposed to protection of a user within the system. In most cases it would be trivial but for instance in military applications would it not be safer to disallow access to the source code?

Is it possible to encrypt the source code of a kernel?

Thanks again
and thank you for answering my strange and somewhat annoying queries

eskimo456 wrote:would it not be safer to disallow access to the source code?

I feel that the access to the source code should not compromise the security.

Not having the sourcecode has never deterred hackers from breaking in. In fact, doing it for a popular system has the adverse affect of making it popular business (think windows exploits and ios jailbreaks).

Do the hackers not gain access to the source code of windows or IOS (obviously not the original code but an interpretation)? If someone does gain access to something similar to the original code then surly they could modify the OS to allow different applications to run in ring0 and allow for malicious software to then access what it wants within the system?

Obviously my knowledge of hacking an OS very lacking and my assumption may be way off
Thanks again

Think of the source as having a map of a military base. Sure you now know where everything is, but it doesn't really help you get past the guards (but might tell you where there is an unguarded entrance).

People who do iOS jailbreaks and other exploiting of "closed source" systems do so either by experimenting, or by looking at the compiled code (which might not be as easy to read as the source, but to those with expirence it can be as useful)

Many applications attempt to obsfucate their machine code (by having redundant code, encrypting it and using a decryption stub, using less optimal algorithms that are harder to understand once compiled) but this cannot stop a dertermined person.

There is almost no way to stop someone reading your code once it is out in the wild. Any technique of encryption can be circumvented by going a layer lower. (Encrypted binary, just use a debugger. Filesystem permissions, reboot into a live CD. Hardware encryption, still circumventable)

The only way to keep your code secure is to run your own data centre and make people log into it from the outside to use your software without ever getting direct access to the code. All you have to do then is make sure no one can hack into it, and you'll have to stand guard over it with a machine gun twenty four hours a day to make sure no one working for you steals copies to sell to your rivals. One little mistake and it's all gone.

eskimo456 wrote:Hi there
I was wandering is it possible to encrypt an OS and prevent people from accessing code. If the OS is written in assembly what stops another user from simply putting the OS through a disassembler and gaining access to the Kernel code? Is it possible to use some kind of encryption method?
I just cant get my head around how that would work since most things that use some kind of encryption use the OS to access hardware etc. How would the OS be able to input commands the the hardware would understand whilst also being encrypted, surly it would need the CPU to be able to decrypt itself?

Many thanks

As previously said before, if you have a ICE or JTAG debugger, you can always inspect the memory if the hardware allows it.

I've seen implementation in practice that use ARM TrustZone in order make more more secure. TrustZone is basically a mode in the ARM where you can only execute code that is considered secure. TrustZone also applies for peripherals on the bus so if you have some kind of security hardware block, you usually make this block only accessible in secure mode.

Let's say you have some kernel code you really don't want anybody to see. What you usually do is having this code encrypted in the storage. During boot you let a hardware block do the encryption of you code and you also put it the secure area. Now it is impossible for user programs and the kernel to even look at it. Services to the secure code are done with a special version of the system call instruction.

This type of security is often used in cell phones (which are hacked regardless).

eskimo456 wrote:I am more curious in protection in terms of an intellectual property/copyright perspective as opposed to protection of a user within the system.

In that case, there are legal actions to do - historical records show that any attempt with technological way in order to prevent abuse of intellectual property end up failed.
You may think, oh I don't have a legal department, but then the software business we talking about is so tiny that there is little threat of copyright issues.

eskimo456 wrote:Thanks for the fast reply.
So in the case of an OS similar to windows were it would be directly installed on the machine this would be near on impossible to actually secure. Thats what I thought. In theory could you have an OS that did the hardware and acted as a main Kernel. Then secondary to this have another OS running in a virtual machine? Could the secondary OS then be encrypted using the virtual machine?

Is this getting to a stage of theoretically you could, however it would take far to long to do, and would still be possible to hack if a groups of people wanted to such as lulzsec, annonymous, Rarzr1911 etc.

Obviously for my own little OS that I've started working on it wouldn't be worth having it if I gave it for free let alone hacking it .
Its mainly a theoretical question just to satisfy my curiosity.

check out Qubes OS

as for encryption, why not make your core files dynamically, randomly, reorganize and temp-password-encrypt themselves and feed the passwords to the kernel which has its own isolation/obfuscation of some sort?