OSDev.org

Hi, I was wondering if there's a way to protect against kernel stack overflows in protected mode.

For my long mode kernel I just use ISTs with one guard page for stacks and it works like a charm, however, ISTs aren't a thing in protected mode
so the protected mode kernel just triple faults after a stack overflow (which is still better than overwriting other random kernel memory).
Is there anything else I could do to force it to use a different stack for page fault/double fault handler? I know there are task gates or something like that, but I'm not sure
how they work, and if they're even a viable option for this.

Thanks.

Task gates are your best bet. I implemented them in a couple hours. They are more complex then the IST, but still doable. Look in the Intel SDM section on Task Management. Basically, all you do i setup a TSS in the GDT like normal. The TSS's EIP must be set to the entry point of your handler, esp should be set to the top of the stack you wish to use, SS, CS, and DS should be set to their respective segments, and CR3 should be set to the page directory you want to use. You then put in the GDT. It should have a type 0f 0x05, and then where you normally put CS is where the TSS's descriptor goes. You also need to have a different task in TR before the double fault occurs. Reading the saved state isn't that difficult, you just find the back linked TSS and read the values from it, as the previous tasks's state is saved. But read the manual first before doing this. An example of this is at https://github.com/Nexware-Project/micr ... i386/pdr.c

nexos wrote:Task gates are your best bet. I implemented them in a couple hours. They are more complex then the IST, but still doable. Look in the Intel SDM section on Task Management. Basically, all you do i setup a TSS in the GDT like normal. The TSS's EIP must be set to the entry point of your handler, esp should be set to the top of the stack you wish to use, SS, CS, and DS should be set to their respective segments, and CR3 should be set to the page directory you want to use. You then put in the GDT. It should have a type 0f 0x05, and then where you normally put CS is where the TSS's descriptor goes. You also need to have a different task in TR before the double fault occurs. Reading the saved state isn't that difficult, you just find the back linked TSS and read the values from it, as the previous tasks's state is saved. But read the manual first before doing this. An example of this is at https://github.com/Nexware-Project/micr ... i386/pdr.c

Thanks!