Strange behavior with Linux binaries

[Tomaka17]

Hello,

I was wondering if someone had already tried executing binaries that were originally compiled for Linux?

If so I've got a few questions

1) How is errno handled? By examining the Linux kernel code I'm about 90 % sure that when an error occurs, the errno value is returned negatively in the EAX register (and then errno is updated by the function which made the syscall) ; what makes me doubt is that official docs say for almost all functions "on error it should return -1 and update errno"

2) How does sys_brk work? According to the docs it should just store the parameter (ie. the end of the heap), returning -1 if the parameter is too large and 0 otherwise ; but with some experimentations I discovered that the Linux kernel doesn't update the value if it is null, and always returns the stored value

3) What docs do you use? Official manual seems to say incorrect things and the Linux kernel is really messy, I lose a lot of time looking for the function I want in the kernel. Variables and structs with non-explicit names doesn't help

I've already found some useful links:
http://foosec.pl/pub/info/syscalls_linux_2_2.html (did Linux 2.2 only have 190 syscalls? 2.6.29 has 332 ones)
http://webster.cs.ucr.edu/Page_Linux/LinuxSysCalls.pdf
http://linux-documentation.com/en/man/man2/
http://linux.die.net/include/asm/unistd.h
But none of them is really good

For example I'm currently stuck on what I should return for sys_mmap2 with length = 0
According to this page since 2.6.12 the call should fail if length = 0 but if I do so the program I'm trying to execute exits after printing an error message

I'm certainly going to solve this problem soon by myself, but I'm regularly stuck because of lack of actual documentation

[yemista]

I agree with you that the kernel is hard to figure out unless you already know it. Have you heard of the book Linux Core Kernel Commentary? Its out of print, and not real in depth, but its very good at giving you an overview of how things work.

[skyking]

Hello,
1) How is errno handled? By examining the Linux kernel code I'm about 90 % sure that when an error occurs, the errno value is returned negatively in the EAX register (and then errno is updated by the function which made the syscall) ; what makes me doubt is that official docs say for almost all functions "on error it should return -1 and update errno"

The docs says exactly that, the error code is returned in the eax register. If you think about it how should it otherwise return it? There is no reason why the kernel should require that the caller is an exec with the errno variable exposed in the symbol table (which is needed for the kernel to modify it), and even then it would be problematic to report the error when it failed to locate the errno variable.

It is the libc API that returns -1 when error occurs.

[Tomaka17]

The docs says exactly that, the error code is returned in the eax register. If you think about it how should it otherwise return it?

I read that the return value is passed by eax, but I didn't read anything about the errorcode The "function should return -1 and update errno" was confusing me
But as a possible alternative I was thinking of something like a static offset (eg. 0x80000000 + pthread id * sizeof(errno))

Have you heard of the book Linux Core Kernel Commentary? Its out of print, and not real in depth, but its very good at giving you an overview of how things work.

I have never heard about it
In fact I was not really looking for a document about the Linux kernel structure but one about the Linux syscalls
(though I think I won't find the second one, so I'll have to cope with the first one)

[skyking]

The docs says exactly that, the error code is returned in the eax register. If you think about it how should it otherwise return it?

I read that the return value is passed by eax, but I didn't read anything about the errorcode The "function should return -1 and update errno" was confusing me

I don't see where you get this from. If you read section 1.1 in the LinuxSysCalls.pdf you clearly see how syscalls are called and how error codes are returned, I see no mentioning of returning -1 and update errno.

Don't confuse syscalls with posix API functions. They need not be the same.

[JamesM]

The docs says exactly that, the error code is returned in the eax register. If you think about it how should it otherwise return it?

I read that the return value is passed by eax, but I didn't read anything about the errorcode The "function should return -1 and update errno" was confusing me

I don't see where you get this from. If you read section 1.1 in the LinuxSysCalls.pdf you clearly see how syscalls are called and how error codes are returned, I see no mentioning of returning -1 and update errno.

Don't confuse syscalls with posix API functions. They need not be the same.

The point he is aiming at is that linux uses a tiny wrapper around some syscalls (lstat, stat, brk, etc) that convert the parameters and return value into something understood by the linux kernel. Return values -1 .. -4096 are reserved and indicate an error - the user space component then inverts this value, stores it in errno, and returns whatever error code POSIX requires for that function (usually -1, but sometimes 0).

Correct, syscalls and posix API functions need not be the same, but in UNIX kernels the wrapper is so slight for many of these functions that it is easy to forget that it exists at all.

[Tomaka17]

The point he is aiming at is that linux uses a tiny wrapper around some syscalls (lstat, stat, brk, etc) that convert the parameters and return value into something understood by the linux kernel.

Yes that's what I didn't know
I thought that libraries were directly using "int 0x80" and doing the conversion from the returned eax to errno/return value


But I've got another problem
I've been working on it for the whole afternoon without a clue

When I call (with Linux) 'strace -i testprog' I'm getting this:

Code: Select all

[b7ff691b] brk(0)                       = 0x97ec000
[b7ff71c1] access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
[b7ff7313] mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fdf000

I see that mmap2 (certainly the wrapper function) is called with length = 8192, flags = 0x22, start = NULL

But when I execute 'testprog' on my O/S, this is the log file:

Code: Select all

Handled syscall: 0x7A (sys_uname, retreives informations about the system)
Handled syscall: 0x2D (sys_brk)
BRK with 0x0, returning 0x804A000
Handled syscall: 0xC0 (sys_mmap2)
Anonymous MMAP Length: 0x0 ; returning 0x804A000 ; flags = 0x22 ; start = 0x0
Handled syscall: 0x21 (sys_access)
Testing access to /etc/ld.so.nohwcap
Handled syscall: 0xC0
Anonymous MMAP Length: 0x0 ; returning 0x804A000 ; flags = 0x22 ; start = 0x0
Handled syscall: 0xC0
Anonymous MMAP Length: 0x0 ; returning 0x804A000 ; flags = 0x22 ; start = 0x0
Handled syscall: 0xC0
Anonymous MMAP Length: 0x0 ; returning 0x804A000 ; flags = 0x22 ; start = 0x0
etc.

(things in parenthesis are my additions, they were not in real logs)

As you can see I'm having troubles with the mmap2 syscall
Everything seems correct except that it is called multiple times and that the length is always 0 (I'm not printing the file descriptor or the protection flags but I checked them and they are correct too)

I don't understand the problem because all the parameters of all the syscalls are correct except mmap2's length
The problem is not about retreiving the second parameter of a function (contained in ECX) as the sys_write function is working (see below) and the printed string is in its second parameter

I compiled the same "Hello world!" program statically and the log is:

Code: Select all

Handled syscall: 0x7A (sys_uname)
Handled syscall: 0xC9
Handled syscall: 0xC7 (0xC8-0xCA are about getting uid/gid)
Handled syscall: 0xCA
Handled syscall: 0xC8
Handled syscall: 0x2D (sys_brk)
BRK with 0x0, returning 0x80B2000
Handled syscall: 0x2D (same)
BRK with 0x80D3000, returning 0x80D3000
Handled syscall: 0xC5 (fstat called on stdout)
File descriptor = 0x1
Handled syscall: 0xC0 (sys_mmap2)
Anonymous MMAP Length: 0x8048000 ; returning 0x80D3000 ; flags = 0x22 ; start = 0x0
Handled syscall: 0x4 (sys_write called with fd=stdout, buffer=Hello world!)
Hello world!
Handled syscall: 0x5B (sys_unmap)
Handled syscall: 0xFC (sys_exit_group)
Handled syscall: 0x1 (sys_exit)

Same problem: everything is correct except that mmap2's length is wrong ; this time it requests a length of 0x8048000 (exactly the base address of the program)
As an extent I could say that mmap2's length parameter is always the base address of the binary (either 0x0 for the dynamic linker in the first case, or 0x8048000 for the second one) but I don't ******* know why

Is there something I'm not getting?