Stack or linker issue in execution of large binary in my OS

[iman]

Dear.

I have a small OS supporting no user mode. very code is run in kernel space.
I have 3 GDT entries as NullSegmentSelector, CodeSegmentSelector, and DataSegmentSelector. Code and data segments are overlapped with base = 0, and limit = 0xFFFFFFFF.

In my loader, I have already reserved 100 MB for stack. Not until recently that I started ti implement a custom JPEG decoder for my OS, everything was fine. By everything I mean, US3.0 support, SVGA support, AHCI, FAT32 filesystem support, etc.

This JPEG decoder uses a huge amount of stack (based on SSE and SIMD operations) to be able to work optimized.
I already tested it on my Windows visual studio and all bugs and errors have been addressed. Now that I want to translate it for my OS, I figured out that the size of this JPEG decoder binary is somehow big. From where I know that? In visual studio C++, I had to manually increase the size of stack for my decoder to something like 10 MB. There I had no problem. But now, as soon as my OS reaches to the first function in JPEG decoder, it restarts itself.

I tried to test it on QEMU. Also on qemu, I cannot get it worked and I always get "executing code outside RAM or ROM" fatal error.

What I have done so far:
- try to increase or decrease the stack size in my loader.asm file, but did not help.
- try to use "align 16" for stack bytes in loader.asm, but no way.
- specifically calling -Wl,--stack,10485760 in my gcc command in order to assign a larger stack to the binary file, but again it failed (I thought this might help like what I had done in visual studio to manually increase the stack size).
- using --nmagic flag in my linker.ld file, but it still fails.

The last option, i am not sure about and I found it somewhere saying that perhaps the linker using 4096 aligned files and it might be the source of issue, but I do not know about it. This is my linker:

Code: Select all

ENTRY(LOADER)
OUTPUT_FORMAT(elf32-i386)
OUTPUT_ARCH(i386:i386)
phys = 0x00100000;
SECTIONS
{
    .text phys : AT(phys)
    {
        code = .;
        *(.text)
        *(.rodata)
	. = ALIGN(4096);
    }
    .data : AT(phys + (data - code))
    {
        data = .;
        *(.data)
	. = ALIGN(4096);
    }
    .bss : AT(phys + (bss - code))
    {
        bss = .;
        *(.bss)
	. = ALIGN(4096);
    }
    end = .;
}

I appreciate if someone can give me some clue.

Best regards.
Iman.

[nullplan]

QEMU's error message means you managed to execute memory that was neither RAM nor ROM. This means, either control flow is being diverted to where it really shouldn't go, or, more likely, you have a stack buffer overflow.

Have you tried using QEMU's debugger to find out how you're getting there?

[iman]

QEMU's error message means you managed to execute memory that was neither RAM nor ROM. This means, either control flow is being diverted to where it really shouldn't go, or, more likely, you have a stack buffer overflow.

Have you tried using QEMU's debugger to find out how you're getting there?

It's stack buffer overflow. But I do not know how to handle this. Since I have a lot of globally defined date and other data on stack and there is no other ways to reduce the size of data on stack.

So my main question again comes. Is there a way to increase the stack size (even if I reserved 10 MB as my kernel's stack buffer)?

[nullplan]

If it is a stack buffer overflow, no amount of stack space will solve the problem. Unless you mean a stack overflow, which is a different thing.

Stack buffer overflow: Trying to write 20 bytes in a stack buffer that's 10 bytes long. Solution: Either make the buffer longer or make the input shorter. Or use truncating code or something.

Stack overflow: Trying to allocate more space on stack than is available. Solution: Make the stack larger.

No amount of linker or compiler flags will fix the actual problem with the stack. If it is a stack overflow, then how do you set up your stack? Where does it actually come from?

[iman]

If it is a stack buffer overflow, no amount of stack space will solve the problem. Unless you mean a stack overflow, which is a different thing.

Stack buffer overflow: Trying to write 20 bytes in a stack buffer that's 10 bytes long. Solution: Either make the buffer longer or make the input shorter. Or use truncating code or something.

Stack overflow: Trying to allocate more space on stack than is available. Solution: Make the stack larger.

No amount of linker or compiler flags will fix the actual problem with the stack. If it is a stack overflow, then how do you set up your stack? Where does it actually come from?

After your explanation, I should say that I meant stack overflow and not stack buffer overflow.
In the beginning, in my loader, I assigned enough stack space for kernel. Is this different than space necessary for actual codes? If so, I have also the code segment space.

Where exactly is the needed space to load the final binaries ?

[nullplan]

Well, stack handling is a little like voodoo. Most OSes will allocate some amount of stack space (large enough for arguments and environment variables and whatever else is on the stack) for the process, and if a page fault happens to be close to the start of stack, it is automatically expanded.

This is the biggest difference between the stack of the primary thread and the stacks of all the other threads in a multithreaded Linux program. The primary stack is allocated by the OS whenever needed. The other stacks are allocated by userspace on thread creation. Thus the primary stack allocation is fragile, and the other stacks are much safer.

From what you wrote earlier, it seems likely to me that you have in your loader file code like this:

Code: Select all

section .bss
boot_stack: resd 1024
.end:
section .text
[...]
mov esp, boot_stack.end
call kmain

Or something like it. That is your stack size.

[iman]

Well, stack handling is a little like voodoo. Most OSes will allocate some amount of stack space (large enough for arguments and environment variables and whatever else is on the stack) for the process, and if a page fault happens to be close to the start of stack, it is automatically expanded.

This is the biggest difference between the stack of the primary thread and the stacks of all the other threads in a multithreaded Linux program. The primary stack is allocated by the OS whenever needed. The other stacks are allocated by userspace on thread creation. Thus the primary stack allocation is fragile, and the other stacks are much safer.

From what you wrote earlier, it seems likely to me that you have in your loader file code like this:

Code: Select all

section .bss
boot_stack: resd 1024
.end:
section .text
[...]
mov esp, boot_stack.end
call kmain

Or something like it. That is your stack size.

Dear nullplan.
I monitored my memory sections. These are the info that I got:
.text
BEGIN=0x100000, END=0x1126E8, size=78KB
.data:
BEGIN=0x114640, END=0x1606AC, size=314KB
.bss
BEGIN=0x161000, END=0x25EC68, size=1.04MB

I have read in OsDev wiki, in the article about Triple Fault (this link: https://wiki.osdev.org/Triple_Fault) that a frequent cause of triple faults is a kernel stack overflow. If the stack reaches an invalid page (one with its present bit clear), a page fault is generated. However, the CPU faults while trying to push the exception information on to the stack, so a double fault is generated. The same problem still exists so a triple fault is generated.

Even though I have a large reserved memory as my kernel stack (1MB), since my OS does not support paging, is it possible that I get my OS restarted because my .text, or .data, or .bss generates a page fault?

If so, how can I address it?

[Octocontrabass]

Even though I have a large reserved memory as my kernel stack (1MB), since my OS does not support paging, is it possible that I get my OS restarted because my .text, or .data, or .bss generates a page fault?

No. Since you are not using paging, nothing can cause a page fault.

But what happens if you need more than 1MB of stack space anyway? Your stack will keep growing towards lower addresses, overwriting everything in its way. Eventually it will overwrite something important and cause an exception. What if it overwrites the GDT or IDT? Well, with a corrupt GDT or IDT, that exception will most likely cause a double fault, and since the GDT or IDT is still corrupt, that double fault will cause a triple fault.

This JPEG decoder uses a huge amount of stack (based on SSE and SIMD operations) to be able to work optimized.

What does it do that uses so much stack? Perhaps it would be better to have dynamic memory allocation like malloc() instead of using the stack for everything.

[iman]

Even though I have a large reserved memory as my kernel stack (1MB), since my OS does not support paging, is it possible that I get my OS restarted because my .text, or .data, or .bss generates a page fault?

No. Since you are not using paging, nothing can cause a page fault.

But what happens if you need more than 1MB of stack space anyway? Your stack will keep growing towards lower addresses, overwriting everything in its way. Eventually it will overwrite something important and cause an exception. What if it overwrites the GDT or IDT? Well, with a corrupt GDT or IDT, that exception will most likely cause a double fault, and since the GDT or IDT is still corrupt, that double fault will cause a triple fault.

This JPEG decoder uses a huge amount of stack (based on SSE and SIMD operations) to be able to work optimized.

What does it do that uses so much stack? Perhaps it would be better to have dynamic memory allocation like malloc() instead of using the stack for everything.

Thanks for your reply.
You are absolutely right about stack growing downwards and overwriting very important stuff such as GDT or IDT.
This is why I always keep an eye on the amount of stack space required for the codes.

Regarding this specific issue, I finally found the problem. It was the way that I reserved so many data on stack for decoder to run. I modified them to be dynamically allocated and they are good now.

Besides I have been told in another entry in forum (here: viewtopic.php?f=1&t=36277) that to enable SSE, I can safely use fxsave and ldmxcsr.
But I figured out that they generated some faults too. I removed this part during the SSE enabling and now it works as I expected.

[Octocontrabass]

Besides I have been told in another entry in forum (here: viewtopic.php?f=1&t=36277) that to enable SSE, I can safely use fxsave and ldmxcsr.
But I figured out that they generated some faults too.

Did you remember to align your operands? FXSAVE requires 16-byte alignment.

[iman]

Besides I have been told in another entry in forum (here: viewtopic.php?f=1&t=36277) that to enable SSE, I can safely use fxsave and ldmxcsr.
But I figured out that they generated some faults too.

Did you remember to align your operands? FXSAVE requires 16-byte alignment.

Yes, they were 16 byte aligned.