Strange stack overflow

[c0x]

Hi!

I've found a strange "bug" in my kernel and i can't figure out how and why it happens.

When i call snprintf somehow it uses 400.000 bytes from the stack that causes it to overflow. I also write the esp values before and after the call to serial port to catch debug information.

The caller code is the following:

Code: Select all

        __asm__ __volatile__( "cli\n" );

        __asm__ __volatile__( "mov %%esp, %%eax" : "=a" ( esp ) );

        do {
          m = esp % 10;
          esp = esp / 10;

          while ( inb( SERIAL_PORT( SERIAL_LINE_STATUS_REGISTER ) & 0x40 ) == 0 ) ;
          outb( '0' + m, SERIAL_PORT( SERIAL_TRANSMIT_BUFFER ) );
        } while ( esp != 0 );

        while ( inb( SERIAL_PORT( SERIAL_LINE_STATUS_REGISTER ) & 0x40 ) == 0 ) ;
        outb( '\n', SERIAL_PORT( SERIAL_TRANSMIT_BUFFER ) );

        snprintf( acFullPath, 128, "/dev/disk/%s", sEntry.de_acName );

I also disable the interrupts before this section not to hack up the ESP by the scheduler or other threads

And snprintf code is:

Code: Select all

int snprintf( char* pcBuffer, size_t nSize, const char* pcFormat, ... ) {
  int esp, m;
  int nResult;
  va_list sArgs;

  __asm__ __volatile__( "mov %%esp, %%eax" : "=a" ( esp ) );

        do {
          m = esp % 10;
          esp = esp / 10;

          while ( inb( SERIAL_PORT( SERIAL_LINE_STATUS_REGISTER ) & 0x40 ) == 0 ) ;
          outb( '0' + m, SERIAL_PORT( SERIAL_TRANSMIT_BUFFER ) );
        } while ( esp != 0 );

          while ( inb( SERIAL_PORT( SERIAL_LINE_STATUS_REGISTER ) & 0x40 ) == 0 ) ;
          outb( '\n', SERIAL_PORT( SERIAL_TRANSMIT_BUFFER ) );

...

As you can see from serial port i can get the ESP values if my code is correct. So the values i got are:
- before snprintf call, ESP=4643625
- after snprintf call, ESP=4243625

Those numbers are in decimal format, and the difference between two ESPs is 400.000 bytes!

I also have the disassembled code of the binary. It can be found here: http://phpfi.com/313985

Could you suggest me something where to start fixing it or something? Unfortunately i ran out of ideas

Thanks, c0x

[JamesM]

Hi,

Given your disassembly I cannot see where that stack pointer gets changed so much - the snprintf function only has a stack frame size of 0x14 bytes!

Personally, given the numbers you have shown, I would actually suspect your decimal output routines of giving you incorrect information.

Cheers,

James

EDIT: Either that or... How large are the parameters you're passing to snprintf?

[c0x]

Parameters should be passed by pointer instead of values, so i think that should not be a problem...

I just noticed one more thing after posting my question here. The numbers (ESP values) written to serial port are reversed, so i should read them from right to left

In this case those numbers are much better but still not correct

c0x

[JamesM]

Which means the difference is 40 bytes - that sounds about right to me?

[c0x]

Which means the difference is 40 bytes - that sounds about right to me?

Yeah, 40 bytes is much better. I also printed the two endpoint of the stack region from the current thread where the above posted code runs. The reversed ESP values are correct stack values then.

The funny thing is the following: I validate my kmalloc structures and lists by asserting magic numbers and before the snprintf call those structs and lists are correct, the validation passes. After the call, in snprintf, between ESP print and other instructions, kmalloc validation fails. Interrupts are disabled so i have no idea what the hell corrupts those structures

That's why i started debugging and thought that the stack is overflowed

c0x

[Solar]

Parameters should be passed by pointer instead of values, so i think that should not be a problem...

I am not exactly sure what you meant to say here, but be aware that C and C++ do call-by-value, not call-by-reference (unless you tell them to explicitly).