pat wrote:Have you recently added a new feature to your OS, or accomplished some milestone?
Over the course of two weeks spent at a small vacation home at the North Sea coast, I spent quite some hours working on
PDCLib, my C standard library implementation aimed specifically at compliance, minimalism (i.e. no "extensions"), and adaptability to your OS of choice. The project, for those who don't know, is what's left of my own OS project from the early 2000's, and explicitly geared toward the OSDev crowd (although I learned it has been adopted in a couple other places as well, including an inofficial XBox SDK, which I think is awesome).
The things I achieved in these two weeks, aside from recharging my batteries and spending some quality time with wife and kids:
- Improving the recent switch to CMake as PDCLib's build system;
- Implementing symbol visibility (i.e., __attribute__("hidden") / dllexport);
- Integration of dlmalloc; PDCLib now has decent memory management.
- Implementation of C11's <threads.h> as a wrapper to Pthreads.
The very stop-gap memory allocation that had been part of PDCLib since 2005 had been a sore point for me ever since. I always intended to, eventually, include dlmalloc or some other PD memory allocator, but never actually got around to it. Erin Shepherd did it while she was maintaining PDCLib, but it was an older version of dlmalloc, and I could not really make heads or tails out of the modifications / config settings that had been done, so I could not adapt that solution easily to my branch of the library.
(*)
What I did now was taking the lastest version of dlmalloc (2.8.6), and "plug it in" to PDCLib. (Both dlmalloc and PDCLib are CC0 licensed, so that wasn't an issue.) It now resides in functions/_dlmalloc, the underscore of the subdirectory indicating that this code is "internal" (like _PDCLIB), not implementing a specific header (the other directories being string, stdlib, locale, you get the idea). Aside from the first couple of lines, in which I make some configuration defines, includes and whatnot, I aimed for leaving Doug's source as untouched as possible. Then I created a
diff file which I added to the repository, so that anybody can see at a glance what changes I made to v2.8.6 of Doug's malloc.c, enabling people to easily apply the same changes to upcoming versions, or different forks of Doug's work.
There are some subtle things to fix some compiler warnings I got with the rather strict warning settings employed by PDCLib's build system. I defined USE_DL_PREFIX, making all the functions use the dl prefix (dlmalloc, dlfree, dlcalloc, ...). I also forwarded DLMALLOC_EXPORT to _PDCLIB_LOCAL, so that all of dlmalloc's internals remain invisible from the outside. I then "unmasked" the standard functions...
Code: Select all
#define dlmalloc malloc
#define dlcalloc calloc
#define dlrealloc realloc
#define dlfree free
#if __STDC_VERSION__ >= 201112L
#define dlmemalign aligned_alloc
#endif
...and
commented out their declarations in malloc.c, relying instead on the _PDCLIB_PUBLIC declarations in PDCLib's own <stdlib.h> (making those 4 / 5 functions visible to clients).
This adds some requirements on the hosting OS (dlmalloc makes use of a variety of system calls), but I feel it's a
massive improvement over my my own stopgap implementation (which used only brk / sbrk, but never gave memory back to the OS until the process terminated, fragmented like crazy and was all kinds of ineffective).
----
I then went ahead and implemented
<threads.h> as a wrapper to pthread.
<threads.h> isn't part of the C99 standard, having made its appearance only in C11 (which I so far considered out-of-scope for PDCLib v1.0). But I perfectly understood the various requests to have PDCLib made thread-safe. The easiest, and most portable, way to do so was to implement <threads.h> and then use
those facilities to get thread-safety (most noteably in <stdio.h>).
The implementation was actually pretty easy. C11's <threads.h> is modelled after pthread anyway, only differing in the types (e.g. thrd_t instead of pthread_t, mtx_t instead of pthread_mutex_t and so on) and the names of the functions.
A snag was that no PDCLib header must actually include <pthread.h>. The user's namespace needs to be kept clean, so dragging in all the identifiers declared by <pthread.h> was a no-go. The solution I ended up with was a helper program,
pthread_readout.c, which -- compiled separately -- includes <pthread.h>, takes its definitions, and uses them to generate the lines of code that need to be included in _PDCLIB_config.h to ensure that PDCLib's data types and pthread's data types are actually cast-compatible, and can be used without having to marshal / unmarshal them on every function call.
Adapting to other threading API's should be possible, either by similar direct-cast compatibility, or more involved marshalling / unmarshalling.
Once that had been done, I went through <stdio.h> and added conditionally-compiled mutex locks to the FILE structure, so that thread safety can be had. (Conditionally compiled, as a platform might not
have threading support, and I wouldn't want to inflict compilation failure on such a platform. Remember, PDCLib is aimed at you lot, so it's "making do" with what's actually there.) The most difficult part was, again, figuring out all the bold assumptions made in Linux space (GCC making statements on the standard library's capabilities, for example, by predefining __STDC_NO_THREADS__ as glibc eschews <threads.h>)...
What remains to be done is shoving errno into thread-specific storage, as required by the standard. That sounds rather easy, until you get to the point where you have to
initialize that on process startup... at this point, PDCLib doesn't hook into pre-main() stuff at all, but uses
static initialization for all its runtime support. I'd much prefer to keep it that way (easier to interface with the host OS), but my vacation is over and I have to get back into the rut of my 9-to-5 job instead of figuring that one out right now.
Oh, and I realized freopen() is probably broken in more ways than one (doesn't remove closed streams from the list of open ones, for one thing), but that's small fish compared to what's been achieved.
----
With the above modifications, I am very close to the point where I can actually claim to have overtaken the Shepherd branch of PDCLib. That had been irking me to no end -- I applaud Erin for the work she did on PDCLib, but felt I couldn't continue where she left off. So I had forked my own branch at the point where
I left off, leaving PDCLib with an unsupported but more feature-complete (Shepherd) branch, and my supported but inferior branch (trunk/master). Being at the point where I can wholeheartedly can say "use trunk, it's got everything the other branch has and more" is a really good feeling of achievement. No more being of two minds which branch to recommend...
----
(*): Yes I know dlmalloc is no longer considered "the cool sh*t". I know there is ptmalloc, and yet other allocators. I will probably look into making them available as options. For now dlmalloc was a) code I had looked at before, b) code that had been "touched" by the maintainer more recently than alternatives I browsed through at a glance, c) code that's also CC0 (hence does not require additional licensing legalese), and d) so much of an improvement over what was previously in PDCLib that merits relative to other available allocator implementations didn't really matter.
. Yes, the page cache is for pages originating from files (it's basically the pages that get mapped to userspace on mmap()). I do not have swapping yet, but the mechanisms will probably be similar. Right now, only the kernel locks memory, but I will also make this available for userspace drivers and to implement mlock().