Porting Newlib: There is always a catch

[Muneer]

Hi,

I have setup multitasking, stdin/out/err interface for user-mode apps and loading and executing of elf executables. So I thought now is a good time to port newlib to my os. I have written all the system calls required by newlib in my kernel and have setup the system call handler to call the function. I have also written the other side of this newlib "essentials" which must be integrated into the newlib. I named it "syscalls.c".

Now I have been following the wiki for porting newlib. Mainly it has the "Porting Newlib" and "Os Specific Toolchain" article. After following the "Os Specific..." I managed to compile newlib after loads of errors creeping up in the form of "makeinfo missing", "i586-elf-cc" not found and finally a weird "i586-elf-ranlib" not found which I can find but make cannot. Think there is a problem with the make. Anyways corrected it after editing the makefile... Now to the point.

* I didnt compile another gcc cross compiler becoz I already had one working right now. Is it a problem?
* I got the newlib header files and the library libc.a, etc..

But how would I use it with the present compiler. please dont ask me to compile gcc. It takes loads of time to compile in this crappy system...

I tried to compile my user apps WITHOUT newlib and WITH the system call interfaces like "write" and it works perfectly.

But now I try to compile my apps with newlib header files and somehow links it with my own linker script. I already wrote one crt0.o startup script same as in the wiki. But when I compile and run it seems as if the printf doesn't get called but the rest goes on. Also once or twice I got undefined references to most of the system calls when I tried to link with libc.a ... I smell something fishy.. I havent got the clear concept here. Asking for guidance

[orafy]

There must be something wrong with your newlib or your LD_FLAGS.
And a dirty way to overcome the "undefined reference to XXX" error:
Append your syscall object to linker, eg:

Code: Select all

 target-machine-ld userprog.o syscall.o -lc -lm  -o userprog 

[Solar]

Check the GCC configuration docs, specifically the section "Cross-Compiler-Specific Options".

In summary, you specify where the cross-compiler should look for system include files and libraries during the ./configure step. You remember you specified "--without-headers" there? That's where you should specify "--with-sysroot=... --with-newlib" now.

Perhaps you could patch your includes and lib into the cross-compiler directory structure retroactively, but I never tried that myself.

The errors that popped up while compiling newlib would worry me, though. You should be able to configure the dependency on makeinfo away, and exporting CC=i586-elf-gcc should solve the "i586-elf-cc not found" issue. Generally speaking, if you have to touch third-party sources, that's either grounds for a bug report to upstream or an error in your approach.

[Muneer]

@orafy

I was testing with newlib and compiling for a dozen times now... I now tried the approach in "Porting Newlib" wiki page. I replaced the /libgloss/write.c and to no success. But when I tried replacing only the _write function in /libglosss/libnosys/write.c with _write function and removing the "stub warning" in there, the undefined references problem was solved. I replaced _write with inline assembly and when I checked it with kernel, I could figure out the system call was made and the "file" arg, and "len' arg are correctly placed but kernel sees the ptr arguement as null.. I suspect my gcc inline code(I really am not used to it). but it has been tested to work without newlib. I was testin the relative equivalents of the ld command-line script you posted.



@Solar

Yes I do remember the "--without-headers" option and what it stood for. And thanks for the link I shall check it out.

For the "makeinfo missing" error I tried "echo "MAKEINFO = :" >> Makefile" just before "make".
I created a symbolic link named i586-elf-cc to i586-elf-gcc in the same directory and included the path in ~/.baschrc to solve i586-elf-cc problem... But I dont know why it never solved the i586-elf-ranlib although at the configure stage it could detect it but in "make install" stage the error occurs.. So I edited /build-newlib/Makefile and changed the following to absolute addresses. Why isnt these workarounds mentioned in the wiki.
LIPO_FOR_TARGET=i586-elf-lipo
NM_FOR_TARGET=i586-elf-nm
OBJDUMP_FOR_TARGET=i586-elf-objdump
RANLIB_FOR_TARGET=i586-elf-ranlib
STRIP_FOR_TARGET=i586-elf-strip
WINDRES_FOR_TARGET=i586-elf-windres

Generally speaking, if you have to touch third-party sources, that's either grounds for a bug report to upstream or an error in your approach.

I guess there is no chance for bugs with these errors as most of them are path problems and makeinfo problem has already been reported.

The reason why I consider against recompiling the binutils and gcc apart from my crappy system is that, I just wanna ensure whether newlib "syscalls" work as expected or the whole of the time it takes to compile gcc is wasted.

Perhaps you could patch your includes and lib into the cross-compiler directory structure retroactively, but I never tried that myself.

I guess that would save a lot of time and ensure newlib actually works before doing the compiling.
Since I could get into kernel syscall handler as I mentioned above, I think it would work.
In the newlib sources, there seems to be a dozen redundant files that looks as the syscall handler. Why is it? And which is the one we should actually change. Is it in the /libgloss or /libgloss/libnosys or /newlib/libc/syscalls.

[orafy]

I guess that would save a lot of time and ensure newlib actually works before doing the compiling.
Since I could get into kernel syscall handler as I mentioned above, I think it would work.
In the newlib sources, there seems to be a dozen redundant files that looks as the syscall handler. Why is it? And which is the one we should actually change. Is it in the /libgloss or /libgloss/libnosys or /newlib/libc/syscalls.

My way is to create a new directory under newlib-1.19.0/newlib/libc/sys/ with your os's name, then put the syscall related files there.
Eg:

Code: Select all

cp  orzos-syscalls.c newlib-1.19.0/newlib/libc/sys/orzos/syscalls.c

Then patch build script like below:

Code: Select all

--- ../../newlib-1.19.0/config.sub	2009-11-20 05:10:09.000000000 -0500
+++ config.sub	2010-04-03 15:47:27.000000000 -0400
@@ -1281,6 +1281,7 @@ case $os in
 	      | -kopensolaris* \
 	      | -amigaos* | -amigados* | -msdos* | -newsos* | -unicos* | -aof* \
 	      | -aos* | -aros* \
+		  | -orzos* \
 	      | -nindy* | -vxsim* | -vxworks* | -ebmon* | -hms* | -mvs* \
 	      | -clix* | -riscos* | -uniplus* | -iris* | -rtu* | -xenix* \
 	      | -hiux* | -386bsd* | -knetbsd* | -mirbsd* | -netbsd* \
--- ../../newlib-1.19.0/newlib/libc/sys/configure.in	2007-05-24 13:33:37.000000000 -0400
+++ newlib/libc/sys/configure.in	2010-04-03 15:55:26.000000000 -0400
@@ -44,6 +44,7 @@ if test -n "${sys_dir}"; then
 	sysvnecv70) AC_CONFIG_SUBDIRS(sysvnecv70) ;;
 	tic80) AC_CONFIG_SUBDIRS(tic80) ;;
 	w65) AC_CONFIG_SUBDIRS(w65) ;;
+	orzos) AC_CONFIG_SUBDIRS(orzos) ;;
 	z8ksim) AC_CONFIG_SUBDIRS(z8ksim) ;;
   esac;
 fi

That's why there is many redundant dirs and files under newlib/libc/sys/

[Muneer]

I shall check it out and post the results

[Muneer]

I have ported the newlib and using it by way of the dirty hack mentioned by orafy. I will move on with this for the moment and will come back to gcc & binutils compiling at a later stage.

I checked the printf and it seemed to work. Figured out that it actually depends upon sbrk. Also you have to either fflush(stdout) or insert a "\n" after the printf to be actually printed to the stdout. It seemed that in the printf a system call was made to fstat before the call to write.

Although I have made the stub in the kernel, the kernel just return without even bothering (currently I am lazy)to set the eax of the cur_process and it doesnt fill the buffer with status of the filedes. so that means garbage returned. I thought that newlib was calling fstat as security to ensure the file actually exists before calling write in printf. But printf i.e sbrk and write succeds..

So why call fstat in the first place. Am I missing something or is it the most highly unlikely rarest possibility of having the buf and return value marked with the CORRECT garbage value every time .

EDIT: Also If I insert my syscalls.o into the libc.a then I figure I could just give the location while building gcc.. will it work.
EDIT: I have inserted and newlib works properly. So the same question again. can I give the path while building gcc

[Solar]

Also you have to either fflush(stdout) or insert a "\n" after the printf to be actually printed to the stdout.

stdout is either line-buffered or fully buffered (if the environment can determine that stdout is not interactive). Hence, no linefeed, no output. I.e., that behaviour is fully compliant.

So why call fstat in the first place.

Who knows? I don't think that printf() relying on sbrk() is a smart move, either. (PDCLib printf() does neither, by the way. )

EDIT: I have inserted and newlib works properly. So the same question again. can I give the path while building gcc

Like in --with-sysroot?

[bluemoon]

I checked the printf and it seemed to work. Figured out that it actually depends upon sbrk.

Did you zero'ed the bss?
When I first ported newlib I haven't zero the bss and printf calls sbrk too. But once I zero bss printf did not call sbrk again.

[Muneer]

Who knows? I don't think that printf() relying on sbrk() is a smart move, either

Right. Well I thought those people out there creating all those great products were great coders who will always have the best design and the best move to get something implemented. Guess I am wrong here.

(PDCLib printf() does neither, by the way. )

Knew it, having used once upon a time in my kernel. But for reasons unknown and (never cared why) it broke when I shifted development to ubuntu.. Well my kernel code was semi-crappy at that time. I dunno but I then ported another printf to my kernel that was at hand at the moment.. I still use your malloc in my kernel though.

Like in --with-sysroot?

Yes. That was what I was asking.

[Muneer]

@bluemoon

Well I haven't zeroed the bss. I just wanted to check out newlib as soon as possible. If you are right about bss being zeroed will not call sbrk, then I was wrong above. I had actually forgot about it. I will now try on zeroing the bss. Really happy that I got newlib ported

Next I must implement all of the kernel system call handlers. From there I guess gcc/toolchain and nasm..

[Solar]

I still use your malloc in my kernel though.

That is definitely the wrong part of PDCLib to use. You have read the comment about it being a "primitive placeholder", did you? The allocation algorithm couldn't scale to save its live, free() doesn't, and the whole shebang isn't thread-safe. (Like the rest of the lib, but I'd think twice about thread safety in malloc().)

[Muneer]

That is definitely the wrong part of PDCLib to use. You have read the comment about it being a "primitive placeholder", did you? The allocation algorithm couldn't scale to save its live, free() doesn't, and the whole shebang isn't thread-safe. (Like the rest of the lib, but I'd think twice about thread safety in malloc().)

Yes I have. Knew that your free doesn't return pages to os. Anyways since due to the present status of my kernel like being non-multprocessor capable, non-preemption of kernel itself I think I can safely use it. And I always try not to use malloc in critical areas. In such cases I call my k_map directly which scans through the kernel to find a free page-table entry as I have mapped the whole kernel-space-page-directories at kernel-initialization stage and wasted 1 MB for that. So I guess in the present scenario its quite safe. Or is there still any gotchas.

[Solar]

None that I am aware of.