how to detect a stack overflow/underflow?

[zer0]

how i can detect a stack overflow in assembler?
does an interrupt exist for this in x86 architecture? (i386)

best regards

[proxy]

well, probably the best you could do is have the pages directly before and after the stack be not mapped. This way you can get a page fault in both cases, for an overflow, you may simply choose to grow the stack..

Another technique (which is a bit cheesy in my opinion, but i've seen it) is to put a marker as the last 32-bit word on the stack. This way you can see if it's value has changed (and it shouldn't ever). However, if someone wants to intentionally overflow the stack, they can by simply making sure that they write the same value to that spot as was originally there...

just some thoughts..
proxy

[Ineo@work]

Yes I think the page fault is the best (and only ?) way to do it.
In fact for my os, I was planning to let the stack grow automaticaly... how could I have some kind of policy to avoid unlimited growth ? What do you suggest ? (a hard limit ?)

Thanks.
Ineo

[DennisCGc]

Yes I think the page fault is the best (and only ?) way to do it.

There is another exception , specially made for stack overflows.
Don't know what it is.

[Pype.Clicker]

there's indeed the Stack Fault, which is generated when you try to go beyond the stack segment limit. Page guards works quite well in most cases for user programs, but there are situations where they'll fail.

For instance, if the application requires an array so large that it goes beyond all the guard pages and end up in the "data" section ...

Or more important, the page faults cannot be used easily for kernel stacks (as the page fault itself is using the kernel stack). To kill a kernel-level thread that performs a kernel stack overflow, you *do* need a limited stack segment and you should have the Stack Fault descriptor to be a task gate, so that a fresh task with a fresh stack handles the error ...

[Curufir]

I still maintain that the best solution on x86 is to have stack and data be two different segments. Of course, as has been pointed out before, this makes the memory model non-flat and screws up most C compilers ;D.

[Ineo@home]

[...]
Or more important, the page faults cannot be used easily for kernel stacks (as the page fault itself is using the kernel stack). To kill a kernel-level thread that performs a kernel stack overflow, you *do* need a limited stack segment and you should have the Stack Fault descriptor to be a task gate, so that a fresh task with a fresh stack handles the error ...

It's true. I understand my mistake now.
Actualy I never encountered any stack problem with my kernel as it is a really minimalistic one yet.

Now it seems I have some more work
Thanks.
Ineo

[Pype.Clicker]

I still maintain that the best solution on x86 is to have stack and data be two different segments. Of course, as has been pointed out before, this makes the memory model non-flat and screws up most C compilers ;D.

It is possible to have them 2 different segments and still being consistent with C programming model, for instance by keeping the base of those segments equal, but having the data segment expand-up and the stack segment expand-down, and the data segment including the "active" part of the stack ...