Stack Questions

[~]

Will it be enough 1MB of stack space for a 1-core computer?

Will it be enough 1MB of stack space for a multi-core computer? In this case, and just in case that's enough, would it be good to divide the Megabyte/number_of_cores?


It's easy to tell that it is far enough for a hobby micro-kernel. But will it be enough for a computer under extreme running processes/resources load (whatever it could typically be considered to be, such as the servers at kernel.org or one of the machines working at Google searches)?

If not, what could be the design/programming details to take into account to reserve an efficient stack space?

[jerryleecooper]

I am no expert on this , but I undestand from what you say that you allocate one mb of memory for what, each process in your OS?
That's alot, but what stack is it? USER stack, of KERNEL stack? if it's KERNEL stack, i would say 32kb is far more than enough, if it's user then therés a trick, allocate pages of memory, and the last page, that would be the first page of the stack, mark it as not present, then whenever it gets accessed allocate more pages! you get your stack growing! isnt that marvelous?

[jal]

you get your stack growing! isnt that marvelous?

Indeed. However, stacks shrink as well, so you have to device something to accomodate for that. And prevent page faults around a stack page border. Not to mention that you still have to reserve the virtual address space for the stack, regardless of the actual physical memory you allocate, so there is a limit.


JAL

[~]

In that case I will reserve 1 MB for kernel stack, for up to 32 processor cores, and will reserve a separate stack for each process module/application.

[t0xic]

32 processor cores.

What?

[Brynet-Inc]

32 processor cores.

What?

I've seen a system with 16 cores... I guess he's just thinking ahead.

[Brendan]

Hi,

In that case I will reserve 1 MB for kernel stack, for up to 32 processor cores, and will reserve a separate stack for each process module/application.

You've got 5 choices that I know of:

• 1) Each thread has it's own kernel stack (which costs losts of RAM for lots of threads)
  2) Each process has it's own kernel stack (but you can't really run 2 threads that belong to the same process running on different CPUs at the same time, which sucks)
  3) Each CPU has it's own kernel stack (but the kernel can't be preempted, which makes it impossible to have "kernel threads", and sucks if the kernel does lengthy operations and/or you want low latency/real-time)
  4) The entire OS has one kernel stack (but this is an extremely difficult thing to do - e.g. make the kernel run on one CPU while other CPUs never run the kernel, and it also sucks for scalability/latency).
  5) Use "dynamic stacks" - for e.g. have one "default stack" per CPU, where (on kernel entry) the kernel immediately allocates the next free "dynamic stack" and switches to it. This is an interesting way to get the same scalability as option 1 without costing lots of RAM for lots of threads, but it adds a little overhead to kernel entry/exit and can be complicated

Based on this choice you can do some maths. For e.g. with option #1 using 4 KB per kernel stack and an OS that supports a maximum of 1 million threads you could calculate that you'd need up to 4 GB of space for kernel stacks (which seems a lot, but someone running 1 million threads would probably want a huge 32 CPU server with 1 TB of RAM anyway). Note: I'm talking about the OS design's limits, not the actual limits for any specific computer (someone with an 80486 and 8 MB of RAM running the same OS will never hit the OS design's limits, as they'd hit hardware limits first).

I normally use 4 KB kernel stacks (for a micro-kernel in assembly) and have never had a problem - the kernel only uses half the kernel stack. You'd need to consider IRQ nesting, functions that call themselves, the chosen language/compiler, etc if you're going to minimise kernel stack size though.

Also, it can be a good idea (for testing purposes) to fill kernel stacks with a special value (e.g. 0xFEEDFACE") and then you can check how many of the special values were overwritten by the kernel (e.g. under heavy load) to determine how much stack space the kernel actually used.

@t0xic: I don't intend to rewrite the kernel every time Intel & AMD increase the number of cores, so I normally allow for up to 255 CPUs, as this is the current hardware limit (8-bit APIC IDs). However, lately I've been getting worried that it won't be long before this hardware limit is extended to 12-bit or 16-bit APIC IDs. CPU manufacturers have been doubling the number of cores every 18 months, so for "4 socket" motherboards we'd be screwed in about 8 years.


Cheers,

Brendan