OSDev.org

I'm trying to design safe strategies for long mode applications so they cannot pass pointers to kernel and let kernel do operations the application itself cannot do.. This seems a lot more complex than I first thought.

Some problems:
1. Application could forge pointer to kernel space
2. Application could pass pointers to unallocated memory
3. Application could pass buffers that are only partially valid

For my 32-bit flat applications, most of these issues are solved by limiting the scope of the flat application selector to only cover valid application space. Particularly, this solves the issue of forging kernel-space pointers, and partly also issue #3. Issue #2 is solved in the page fault handler. A draw-back for 32-bit is that faults will occur in kernel or driver on behalf of the application, but the logic is fast and require no validation of APIs at all.

For 64-bit it is quite different. The kernel cannot automatically detect that a pointer is outside of the valid region of the application, but must check it's limits. Not only that, but it must know the size of the buffer, and at least must know that the end of the buffer is also within application space. Two solve #2, it seems more practical to validate each page-table entry that the buffer covers for validity. In fact, the overhead needed when running a 64-bit application under a 32-bit OS seems to be needed even when running a 64-bit application under a 64-bit OS. It doesn't take much longer to validate page-table entries of a buffer compared to validating and copying them to below 4G. A definite advantage of this is that the kernel could refuse to do requests that involve invalid buffers, and that faults no longer would occur in kernel-space on behalf of the application.

How do others solve these issues?

For things other than buffer, I do with handle, which is an index to kernel structure - so the application do not pass pointer on this kind of syscall.
To validate parameter, do a bound check on the index, then check the entry to validate owner.

To pass small amount of info, use the registers.
To pass page size buffer, check the page structure (only one lookup per page with recursive paging)

For most long mode OSes, you can simplify things greatly to on the basis that the kernel lives in the top half and the application in the bottom half.

Owen wrote:For most long mode OSes, you can simplify things greatly to

Code: Select all

buffer_ptr &= 0x0000 7FFF FFFF FFFF;
buffer_sz  &= 0x0000 7FFF FFFF FFFF;

on the basis that the kernel lives in the top half and the application in the bottom half.

I wouldn't want to do that. What I do instead is something like this: (assuming rdi is the parameter)
That's because I would not want to fix a bad address, and also because the first PML4-entry is invalid for the application. While the kernel is not in the upper region, the page-table alias is, and I also plan to put device-drivers there. I also keep one guard PML4-entry at the top because otherwise pointers could wrap to the device-driver area.

Additionally, I don't allow buffers larger than 1G.

bluemoon wrote:For things other than buffer, I do with handle, which is an index to kernel structure - so the application do not pass pointer on this kind of syscall.
To validate parameter, do a bound check on the index, then check the entry to validate owner.

I use a similar technique. In fact, I don't allow structures to be passed to kernel at all, requiring a handle solution when there is a need to pass a structure. However, there is still a need to pass pointers to buffers.

A guard page can be placed before the start of system reserved memory, to eliminate the need to pass the size for buffers that are accessed sequentially. Every API should be prepared to handle exceptions that occur when reading or writing user buffers. If you're checking the PTEs beforehand, you'll need to make sure the memory doesn't get freed by another thread in the meantime, but an address could be swapped out or refer to a memory mapped file, and then you don't know if an access will succeed.

Gigasoft wrote:A guard page can be placed before the start of system reserved memory, to eliminate the need to pass the size for buffers that are accessed sequentially. Every API should be prepared to handle exceptions that occur when reading or writing user buffers. If you're checking the PTEs beforehand, you'll need to make sure the memory doesn't get freed by another thread in the meantime, but an address could be swapped out or refer to a memory mapped file, and then you don't know if an access will succeed.

In my OS, it cannot be swapped out (because that is not supported). The problem of another thread freeing the buffer is harder to handle. It should only happen if the application is buggy, but if it does happen the result could become catastrophic if the page can be reallocated before the syscall is finished.

OTOH, the heap support routine probably would never free pages, so it shouldn't really happen, other than for memory mapped files.