OSDev.org

Some areas in your memory model are expected to be wiped (.bss, for instance), others do not need this (when picking up a page to map a file, for instance) ...
For pages holding sensitive datas, i think it's up to the program that stores sensitive datas in memory to make sure these data cannot be snooped by other process by wiping pages (or giving them to the page cleaning service rather than reinserting them in the free pages pool). It should also make sure the pages are not swapped out (for paranoiac applications that fear a SUed process is hacking the swap file/device ...)

..

But physical pages could get swapped to disk at any time. And how should an application know what is sensitive? I could write a secret document in a word processor. Should the word processor program treat all its data as sensitive? Should I have to click on a "this document is secret" button?

IMHO this places too much load on applications. They would be forced to know how the memory manager works. The OS documentation would need to say, "If you want to store any information which could possibly be secret in memory, make sure you call lock_pages on that memory before writing to it. Note that this prevents that memory from being paged to disk, until you call unlock_pages. You should erase any possibly secret information from memory before calling unlock_pages."

To me, this is unacceptable, particularly when faced with the alternative: zeroing previously used memory before giving it to an application.

..

It's only necessary to clear the page before it's re-allocated. It doesn't matter if freed pages lie around in the system without being cleared since the only way to access them would be to allocate some memory. Therefore, if you're not having a background thread clearing freed memory, the latest you must clear pages is just before they are given to the application.

Would having the zero list and zero thread be more efficient than zeroing on allocation?

2 Pype.Clicker

then it should make sure that it wipes them before returning them to the free pages pool; (skip) an interface to lock pages from being swaped in and out of memory, though).

(besides other reasons given above, what is the process aborts or gets killed or dies from an operation? it is still expected to wipe itself?)
so the alternatives are:
-having each program or runtime clear all memory, or having to worry which data is sensitive (which is only possible in limited cases), and worry about locking pages (and letting processes interfere with the swapping mechanism! (horror)), and what's with termination.. the whole thing seems to me messy enough just to think about.. those few cycles you saved on zeroing will cost you a lot more overall.
--or--
-having it done always, automatically, trivially, by the operating system? (and having a guarantee that your precious data doesn't get handed to a malicious individual on a sliver page)

I agree wih zloba/Tim

chris wrote: Would having the zero list and zero thread be more efficient than zeroing on allocation?

Depends whether the zero thread can keep up with allocations. I use a hybrid approach. The zero thread zeroes as many freed pages as it can while the system is idle. If some app wants lots of memory fast, and the zero thread can't run, and the system has run out of zeroed pages, the allocator grabs a page from the free list instead and zeroes it explicitly.

Cool, I like that.

Now I just need to figure out how to set up my linked list (yes, only one in the beginning )...

zloba wrote:

then it should make sure that it wipes them before returning them to the free pages pool; (skip) an interface to lock pages from being swaped in and out of memory, though).

(besides other reasons given above, what is the process aborts or gets killed or dies from an operation? it is still expected to wipe itself?)

Hmm, well, looks that i have a weird point of view, as usual ... Let's try to make things clearer.

What i have in my design is the so-called "memory-object", which is a system-dependent implementation of well-known interface for managing memory. So actually all your app would have to do is "i need 16MB for a new 'secure-heap' instance". From there, the system will find back the methods for pages allocation, swapping, deallocation, etc. according to the 'secure-heap' policy.
The application doesn't need to know if it has to lock pages or not. It will just reclaim more heap or give it back through a "secure_heap_object::sbrk" call and the memory class will translate this into appropriate system calls to the kernel.

If the application dies, destroying the secure_heap will also imply trashing properly all the pages it was still holding. I see nothing utterly complicated in this.

You could even have a "document-heap" abstraction that will resolve into "normal-heap" or "secure-heap" or "safe-heap" (with a crash-recovery feature) based on current document policy

Now about the "secure document" button, this is just a preference stuff. Think of it as "create all my documents secure by default" (for NSA-only) or "use default document security" or "create all my documents efficient by default".

I think experience has taught us to put in security by default, not by request. If you make security optional, applications won't bother with it. Sure, applications designed around security will bother -- the author of ChangePassword is going to ask for a secure heap -- but any other application could contain information which the user isn't going to want to spread elsewhere.

Now, I'm not about to suggest that all files in the file system should be encrypted, but that's only because the encryption would have significant overheads. But cleaning pages when they're freed is easy to implement, doesn't slow things down (particularly if you've got a zero-page thread running in the background), and fixes some fundamental 'ways into' your system.

Exercise: Browse an executable produced by the DJGPP linker. Look in the gaps between sections. You may be surprised to see stuff that isn't machine code. ld doesn't clean its memory before using it, and DOS doesn't clean memory before allocating it. I usually find junk from symbol tables there; luckily the last application to run on the DOS subsystem was the compiler.

Reference: Tanenbaum and Woodhull, Operating Systems: Design & Implementation 2nd ed, pp.439-440 ("5.4.3 Generic Security Attacks").

2 Pype.Clicker:
imho,
-all process data should be treated as confidential
-the user won't _ever_ notice those microseconds you saved on zeroing.. especially compared to disk latency in the process of swapping and program loading
-it is unlikely to be widely used the way you expect - everything will be declared either secure (making this pointless) or insecure (making it dangerous). with improvement being O(max number of additional pages) over the lifetime of a process (assuming there is any improvement), it just doesn't pay.
EXCEPT
when you are designing a system without privilege separation and with heavy page-reallocation activity

imho, there are better areas to design and optimize

chris wrote: Would having the zero list and zero thread be more efficient than zeroing on allocation?

just for a concept for OSes, as fits the forum:

An operating system should use the system during idle periods so that it can respond quickest when it is in use.

So, if you're reading what's on screen, the OS can do what it wants to do, but if you're installing an app it should not be required to use some of your, at that point, valuable cpu cycles for background tasks.

If you want all pages to be zeroed (yes you do), you can either run it in the background or in the foreground (in this context, that means at request). In the background would make all pages zeroed when you need them without other work, and in the foreground would make all pages being zeroed when you need them. The first moves the work to the background section of time, which is the best to use for it.

Hi,

This is probably off-topic (I apologize), but hopefully the concept I'm about to explain is enough to justify it..

Candy wrote: just for a concept for OSes, as fits the forum:

An operating system should use the system during idle periods so that it can respond quickest when it is in use.

So, if you're reading what's on screen, the OS can do what it wants to do, but if you're installing an app it should not be required to use some of your, at that point, valuable cpu cycles for background tasks.

If you want all pages to be zeroed (yes you do), you can either run it in the background or in the foreground (in this context, that means at request). In the background would make all pages zeroed when you need them without other work, and in the foreground would make all pages being zeroed when you need them. The first moves the work to the background section of time, which is the best to use for it.

IMHO everything that the CPU does can be classified into 3 distinct types of work: reacting, pre-processing and waiting.

Reacting is where something happens that causes the CPU to do work straight away (e.g. the user presses a key, the hard drive completes a transfer, the ethernet card says a packet has arrived). Pre-processing is where the CPU is doing work that doesn't have to be done straight away, but will hopefully improve performance later on. Waiting is where the CPU can't find anything to do (e.g. it's executing the HLT instruction, or it's sent to sleep, or it's polling hardware).

Now a "perfect" OS would spent 100% of it's time pre-processing! Think about this for a bit...

Imagine that you've turned your computer on, you're waiting for it to finish booting and the doorbell rings. It's a good friend with some snacks and a movie that you've been dying to see for ages (Lord of the Rings! Holy cow ). You grab some drinks, some snacks and settle down to watch the movie, completely forgetting that you've left your computer running.

When you finally get back to your computer it's pre-loaded every single application that you might want to use, and they're all running in the background just waiting for you to "start" one. You click on your web browser and it's up on your screen instantly because the video data was already pre-processed ready to blast into the video card. Then you decide to send the browser to this forum and it's on your screen instantly because the computer knows you come here often and it's already downloaded everything and pre-processed the video for it. You decide to compile your OS while your reading some interesting messages on the forum, so you run 'make' and guess what? It's completed instantly because the OS already compiled it and had the new/modified data stored on disk already - all the OS had to do was to commit the directory entries into the cache (in RAM). Then you decide to scan your 120 GB hard drive for bad sectors - you guessed it already, the scandisk utility skips everything that's been checked recently and completes instantly.

Sound a bit too hard to believe? Well I did say a "perfect" OS, and we all know nothing is perfect...

Most OS's actually spend negligable amounts of time pre-processing. With the above scenario windows would spend a few hours doing absolutely nothing at all, then when you start using the computer it'd go flat out trying to get things done quickly (and Linux isn't any better). After all that's why you paid so many $$$ for that fast new CPU isn't it?

Now consider me sitting here typing. What's my CPU doing? I'll tell you - it's spending 99% of it's time doing absolutely nothing, waiting for me to press a key. Why isn't it using the time between keypresses to scan my disks for bad sectors (it's been a while), or running a spell checker in the background? I guess I'll have to sit here and wait a little while when I tell it to spell check in a few minutes. Why isn't it pre-loading Quake for me - it should know there's at least a 50% chance I'll want it ready when I'm finished typing.

Like I said, nothing is perfect, but an OS can do no pre-processing at all or it can pre-process as much as possible. My OS will be pre-processing as much as possible...


Cheers,

Brendan