port a libc or make one from scratch?

[nspiredev500]

All the libcs I found are either entirely linux-focused or don't have full POSIX features.
So is it worth trying to understand how to re-write the system-dependant parts of an existing libc or should I just try to write my own?
I would like to port existing software someday. Mainly GCC, to become self-hosting, and a wayland server.

And either I compiled my newlib wrong or it's just big, but a printf("Hello world!") is ~50KB. The device I'm targeting has only 64MB of RAM,
but I can only use ~10MB of RAM. It's for a TI nspire and I would like to run my kernel in parallel with the normal OS.
The "running in parallel" part is working for now, but I still have the memory restriction.
So if I can't get my libc to be a shared library, that memory isn't very much.

What have you chosen for your libc?

[alexfru]

What have you chosen for your libc?

I wrote my own. No POSIX. Just most of ANSI C and some extra C99 stuff (e.g. no full support for wide chars, floating point environment, something else that escapes me). Maybe you don't have to have POSIX? My library works in DOS, Windows, Linux, MacOS.
But then it is for my compiler, not gcc or clang or my OS.

[XainFaith]

I just recently ported libc11 its not exactly full featured but it was probably one of the easiest one and the only one i managed to port without pulling out my own hair.
I suggest you look into that one if you just need a c lib even if its not all the bells and whistles.

[eekee]

either I compiled my newlib wrong or it's just big, but a printf("Hello world!") is ~50KB.

That's because printf is bloatware. More seriously, it's known to be large to the point where programs written for very small devices should try to avoid it altogether. I don't think 50KB is unusual, Plan 9 on amd64 produces an 80KB binary. (Plan 9's printf is extensible and, I think, has some funky types enabled by default.) Plan 9 would have initially run on devices with 16MB memory or less, and it's never had dynamic linking, so maybe you don't need to worry.

[nullplan]

That's because printf is bloatware.

Oh yeah, there is a lot of history to that. For instance, there is this project: http://www.fefe.de/fgetty/. Which is just mingetty with the printfs stripped away, saving runtime memory by two orders of magnitude.

With static linking, each function that is referenced, directly or indirectly, will be linked into your project. Well, OK, it is each object file referenced, but well-written libraries have only a single function per object file. And printf() by necessity references all sorts of things, most damagingly floating-point output. If you call printf(), you have code in your binary for printing FP numbers in hex format. Even if you never use it, it is still part of the binary.

Runtime cost is even worse. musl has code to output floating point numbers correctly rounded at maximum precision. The algorithm it uses to produce such output requires a large buffer that is allocated on stack. The exact size depends on what a long double is on the target platform, but on a PC, the buffer ends up being around 8k. That is not the bad part. The bad part is, clang will often inline fmt_fp() into printf_core(), causing every single printf() to consume 8k of stack. But also, on PC, the macro PTHREAD_STACK_MIN is set to 2k. Meaning that a thread running with the minimum stack can never call printf() with floating point numbers, or can never call printf() at all.

[eekee]

Those stack/optimization shenannigans! Plan 9 has optimizing linkers. In particular, they're very good at eliminating dead code. So that 80KB binary is in a sense optimized for size... I'm not sure the Plan 9 linkers are good enough to eliminate printf's floating point hex output. printf is essentially a language interpreter, and if I understand right, those are hard to analyze.

Small point of order for anyone reading: .ar format is a container for multiple object files, so don't look at a .ar file and think the whole thing will be linked in.

[nullplan]

I'm not sure the Plan 9 linkers are good enough to eliminate printf's floating point hex output. printf is essentially a language interpreter, and if I understand right, those are hard to analyze.

It is unlikely the linker of all things would be able to do that. Do those linkers analyze control flow to identify dead code? Anyway, in this case, the linker would have to prove that printf() is never called with "%a" as part of the format string, and in general that is as hard as the halting problem. Therefore it is unlikely they could do that, unless the authors of those linkers solved that problem and forgot to obtain their Fields Medals and the complimentary million dollars.

[eekee]

It is unlikely the linker of all things would be able to do that.

It's a weird design: Plan 9's compiler's don't optimize; its linkers do. Golang was like that at first, but it's a bit of a nuisance because the slow work of optimization is done on every link, so they changed it. It doesn't matter so much in Plan 9 because native programs tend to be very small.

Do those linkers analyze control flow to identify dead code?

Yes, but I don't know how far it goes. They can eliminate `if(0){ foo(); bar(); baz(quux()); }`; this is the preferred way of disabling blocks of code in Plan 9, but I don't know what level of complexity causes them to give up.

Anyway, in this case, the linker would have to prove that printf() is never called with "%a" as part of the format string, and in general that is as hard as the halting problem. Therefore it is unlikely they could do that, unless the authors of those linkers solved that problem and forgot to obtain their Fields Medals and the complimentary million dollars.

Ken Thompson is famous, but I don't think he's a medalled millionare, no.

I wonder if it's really that hard, though. If the linker can prove printf is only called with literal strings, (and I don't see why it couldn't,) then it could check those strings for "%a" or whatever. Also to note: The halting problem only applies to computers which have infinite memory and time. Any program on any lesser computer will eventually halt. Some mathematicians have found this actually goes much deeper than this obvious observation, finding that a finite computer can understand much more about itself than an infinite one. They're working with a language called Agda. Annoyingly, Agda is a constantly-changing research vehicle rather than any kind of platform to build on. And I don't think any of this research was done while Plan 9 was being developed, anyway.

[AndrewAPrice]

It is unlikely the linker of all things would be able to do that. Do those linkers analyze control flow to identify dead code? Anyway, in this case, the linker would have to prove that printf() is never called with "%a" as part of the format string, and in general that is as hard as the halting problem. Therefore it is unlikely they could do that, unless the authors of those linkers solved that problem and forgot to obtain their Fields Medals and the complimentary million dollars.

I know this is C, but it wouldn't suprise me if you could do some constexpr/template trickery in C++17 to evaluate the the string literal at compile time.

[nullplan]

It's a weird design: Plan 9's compiler's don't optimize; its linkers do.

OK, so the linker is more complex than I thought and does some of the job I would allocate to a compiler. And even a non-optimizing linker is already plenty complex. Seems like weird design to me as well.

I wonder if it's really that hard, though. If the linker can prove printf is only called with literal strings,[...]

That's why I wrote "in general". Special, simple cases do exist, but in general, the format string can be constructed at run time. Indeed, the format string can depend on data input at run time, and then nothing can be said about it at all at link time. That is not what you should do with it, but it is a possibility. Also, a conversion specifier for printf() can be pretty long, but if the linker already knows enough about printf() to parse the format string then this will likely not be a problem.

I know this is C, but it wouldn't suprise me if you could do some constexpr/template trickery in C++17 to evaluate the the string literal at compile time.

In C++ you can sidestep the issue by not using printf() at all. If you have a library of operator<< overloads taking an ostream as first argument and returning a reference to it, then the linker can just only link in those operator functions that are referenced in the C++ code. Then the hex-float formatting code can be put into operator<<(ostream&, double), and it will not be present in a Hello World program. Unfortunately, it will still be present in all code that just wants to print some floating point numbers (in decimal). How much constexpr will help with that I don't know. Also, using IO manipulators makes the code just mindbogglingly verbose.

[AndrewAPrice]

I played around and ported newlib to my OS.

Merely going from my own functions that were wrapped around syscalls:

Code: Select all

perception::DebugPrinterSingleton << "Hello " << "world " << (size_t)12 << '\n';

To using printf:

Code: Select all

printf("Hello %s %i\n", "world", 12);

Blew up my binary from 2 KB to 29 KB. This was compiled with GCC with -O3 -flto.

[iansjack]

I would imagine that printf can do considerably more than your system call. Does your system call handle all the formatting options of printf?

In a real OS the code for printf would only be loaded into memory once, and the executables would be smaller because the library calls would be relocation entries.

[eekee]

In a real OS the code for printf would only be loaded into memory once, and the executables would be smaller because the library calls would be relocation entries.

Ah, so Thompson and Ritchie never wrote a real OS in their lives. Unless you count Inferno, but around these parts, there's some disdain for OSs of Inferno's class, too.

Sorry, I had to.

[iansjack]

Perhaps I should have said "real, modern".

[eekee]

Yeah, but they wouldn't even put dynamic linking in Golang which went public in 2011. On the other hand, it does seem to be necessary for GUI in the way it's normally implemented. I dream of GUI as a service rather than a library, and I'm tempted to extend that to printf, but that's another story.