Page 1 of 2

malloc (was: Internal Kernel Pseudo-Garbage Collection)

Posted: Wed Sep 03, 2014 11:01 pm
by Wajideu
Seems like you're misunderstanding how memory management works. Garbage Collection is something that implemented by the runtime an application is linked against, not by the kernel itself. In fact, the kernel shouldn't manage memory at all. It should simply provide a function called sbrk that's used to extend the memory of the running application. Libraries like malloc and talloc then provide a wrapper around sbrk for alloc/realloc/free.

This is why malloc.h is not part of the standard C library. Garbage Collection is a language runtime feature, not a kernel feature.

Re: Internal Kernel Pseudo-Garbage Collection

Posted: Wed Sep 03, 2014 11:18 pm
by KemyLand
So I shouldn't implement this in any way, instead only providing the basic interface?

Re: Internal Kernel Pseudo-Garbage Collection

Posted: Wed Sep 03, 2014 11:22 pm
by Wajideu
KemyLand wrote:So I shouldn't implement this in any way, instead only providing the basic interface?
Yeah. Your kernel should only expose a minimalistic interface; and should leave the functionality to the api.

Re: Internal Kernel Pseudo-Garbage Collection

Posted: Wed Sep 03, 2014 11:25 pm
by Wajideu
KemyLand wrote:PS: Userspace programs WON'T use this, only the kernel and drivers. That's also why I call it a pseudo-GC, because it really doesn't collects, but recieves and frees. The proper function is responsible for calling MMGC()
This is what shared libraries are for. Or you can just not use garbage collection for drivers (seeing as how that's probably a bad idea anyway) and just use sbrk itself.

Re: Internal Kernel Pseudo-Garbage Collection

Posted: Wed Sep 03, 2014 11:30 pm
by Wajideu
KemyLand wrote:It's still perfomance-talking okay to use free() every time it's needed inside the kernel?
As I added to my previous post, you can still use sbrk. But that aside, I can't see a case in which you'd ever need memory allocation in a driver. To be honest, it sounds like a really bad idea.

Re: Internal Kernel Pseudo-Garbage Collection

Posted: Thu Sep 04, 2014 2:55 am
by Wajideu
Combuster wrote:Just to make sure people don't get bad ideas:
DaemonR wrote:Seems like you're misunderstanding how memory management works. Garbage Collection is something that implemented by the runtime an application is linked against
Actually, this statement is bogus. Garbage collection is any mechanism that automates the testing of allocations and freeing of memory when it's no longer in use. Commonly it's specifically referring to managed languages with heavy mark-and-sweep style garbage collectors.
I didn't define what garbage collection was, I stated how it was implemented. It's implemented in many different ways for many different languages as part of the language's runtime.

Re: [SOLVED] Internal Kernel Pseudo-Garbage Collection

Posted: Thu Sep 04, 2014 3:12 am
by Combuster
It's implemented (...) as part of the language's runtime.
I pointed it out because that blanket statement is provably false. Can you do garbage collection by any of it's formal definitions in C? Yes. Where in the runtime is it located? Nowhere because it's not part of the language standard.

The difference is that managed languages add the calls to memory management for you, and that lower level languages require you to do that manually.

Re: [SOLVED] Internal Kernel Pseudo-Garbage Collection

Posted: Thu Sep 04, 2014 4:23 am
by Wajideu
Combuster wrote:
It's implemented (...) as part of the language's runtime.
I pointed it out because that blanket statement is provably false. Can you do garbage collection by any of it's formal definitions in C? Yes. Where in the runtime is it located? Nowhere because it's not part of the language standard.

The difference is that managed languages add the calls to memory management for you, and that lower level languages require you to do that manually.
It doesn't have to be standardized in order to be part of a runtime. Malloc isn't standardized, yet it's still linked with the runtime by default. The point here is that garbage collection is a feature implemented and dependent on the language, not to debate whether or not it's defined by the language's standards.

Re: [SOLVED] Internal Kernel Pseudo-Garbage Collection

Posted: Thu Sep 04, 2014 4:36 am
by Combuster
Malloc isn't standardized
Lies. See C89, C99, C11...
yet it's still linked with the runtime by default.
Lies. Freestanding platforms have no malloc.
garbage collection is a feature implemented and dependent on the language
By its widest definition, this is considered garbage collection too. It's a windows API, and hence not part of the C++ language at all.

Re: [SOLVED] Internal Kernel Pseudo-Garbage Collection

Posted: Thu Sep 04, 2014 4:46 am
by Wajideu
Combuster wrote:
Malloc isn't standardized
Lies. See C89, C99, C11...
Not lies, I've already read the standards. Numerous times actually.
Combuster wrote:
yet it's still linked with the runtime by default.
Lies. Freestanding platforms have no malloc.
Not lies, any person who's ever used C knows you don't have to include any extra libraries to use malloc. Aside the point, you just tried arguing with me in the last sentence that malloc was part of the C standard. Make up your mind.
Combuster wrote:
garbage collection is a feature implemented and dependent on the language
By its widest definition, this is considered garbage collection too. It's a windows API, and hence not part of the C++ language at all.
It is dependent upon the language. Would this exact library work with Python? Java? Clisp? Of course not. It was designed around the syntax of C++. You're seem to be misinterpreting what I'm saying. I'm not saying garbage collection is a part of the language, I'm saying it's implemented and dependent upon the language. A C++ garbage collector doesn't work the same way as a C# or Java garbage collector, and they're not cross compatible.

Re: [SOLVED] Internal Kernel Pseudo-Garbage Collection

Posted: Thu Sep 04, 2014 5:04 am
by Icee
@DaemonR
Authority mode on or something?

On malloc(): C11, section 7.22.3. On list of functions available in freestanding environments: C11, section 4, paragraph 6. You might also want to have a look at section 5.1.2 for definition of freestanding vs hosted environment.

Re: Internal Kernel Pseudo-Garbage Collection

Posted: Thu Sep 04, 2014 5:11 am
by sortie
Please forgive me for going a bit off-topic here, but I need to rant.

DaemonR: You are inexperienced, and you are giving inadvisable / irrelevant / ignorant advise. This is harmful to newcomers. Stop doing so until you really know what you are talking about, because as Combuster and I will demonstrate, you don't know what you are talking about.
DaemonR wrote:In fact, the kernel shouldn't manage memory at all. It should simply provide a function called sbrk that's used to extend the memory of the running application. Libraries like malloc and talloc then provide a wrapper around sbrk for alloc/realloc/free. This is why malloc.h is not part of the standard C library. Garbage Collection is a language runtime feature, not a kernel feature.
What the **** are you talking about? Managing memory at the page level is one of the most important tasks of the kernel! The data structure maintained by malloc is done entirely in user-space, but it is constructed in pages provided by the kernel.

Malloc is not a library, it's part of the standard library implementation, i.e. just as automatically-available as strlen in hosted C programs. Talloc is something entirely else, actually a library, but it's _not_ built upon sbrk as that would be highly dangerous, it's either built on malloc or mmap. Don't advise people to implement sbrk, it's a broken obsolete interface that cannot be used safely anywhere but in the malloc implementation, and even there it has serious design flaws such as not allowing the heap to be fragmented. The interface people should be advised to implement is mmap() - you can fairly easily implement a subset of it (no file mapping, just anonymous pages), and you can gradually improve it with more features. You can't do that with sbrk. In fact, it's best to stop talking about sbrk (unless telling people not to implement it) lest newcomers actually implement it (I did - what a waste of time).

You are missing the point. I think the OP is talking about using garbage collection inside the kernel, not providing it to user-space. This is entirely reasonable in principle. There are designs where the kernel could provide garbage collection to user-space, especially if the user-space code is managed. If the user-space code has not been verified, it's probably dangerous to do garbage-collection in the kernel from a security point of view.

Finally, the reason malloc.h is not part of the standard is that there is no reason for it to exist. The standard malloc(3)/free(3)/realloc(3)/calloc(3) routines are defined in the stdlib.h header. If a system has malloc.h it is non-standard and nobody knows to put in there. Some systems put malloc internals there. Either way, programs can't trust that header to do anything meaningful for them and including it is an error. Did you think malloc(3) was defined in malloc.h instead of stdlib.h? Then you really don't know what you are talking about.
DaemonR wrote:This is what shared libraries are for. Or you can just not use garbage collection for drivers (seeing as how that's probably a bad idea anyway) and just use sbrk itself.
DaemonR wrote:As I added to my previous post, you can still use sbrk. But that aside, I can't see a case in which you'd ever need memory allocation in a driver. To be honest, it sounds like a really bad idea.
What are you taking about? Shared libraries are generally a user-space thing (or are you thinking about loadable kernel modules, but still tyhen, what are you talking about). Have you even implemented real drivers? Memory allocations are useful there. Yes, you shouldn't do it in the main interrupt path to avoid deadlocks and have error cases, but it's perfectly reasonable to do memory allocations from drivers. Again, don't recommend code use sbrk, it cannot be safely used from outside of malloc (and malloc should be built using mmap).
DaemonR wrote:It doesn't have to be standardized in order to be part of a runtime. Malloc isn't standardized, yet it's still linked with the runtime by default. The point here is that garbage collection is a feature implemented and dependent on the language, not to debate whether or not it's defined by the language's standards.
Malloc is very much standardized. It's was added to Unix C after calloc proved a bit bothersome because it had two arguments for no real reason (at least, as far as I can see in the early Unix source code). It was standardized as part of C89. “still linked with the runtime by default” - I'm not sure what you mean here (and I fear you misunderstood something) but I see a single interpretation where this is true, so I won't bash that remark.

In short, while there are a few shattered truths in your posts here, they are generally filled with false information and bad advise. Please get more experience by implementing a real operating system, so you actually can be useful to newcomers. There is no reason to reply to my post, it'll only make more noise than good.

Re: [SOLVED] Internal Kernel Pseudo-Garbage Collection

Posted: Thu Sep 04, 2014 5:17 am
by Wajideu
Icee wrote:@DaemonR
Authority mode on or something?

On malloc(): C11, section 7.22.3. On list of functions available in freestanding environments: C11, section 4, paragraph 6. You might also want to have a look at section 5.1.2 for definition of freestanding vs hosted environment.
Why don't you take a second look at that yourself. While there is example code using the malloc function, malloc.h is not included in the list of standard headers, nor is the implementation for malloc defined within the document. As I stated before, I've read the standards numerous times, I didn't even need to look at them to know they're not in it.

Re: [SOLVED] Internal Kernel Pseudo-Garbage Collection

Posted: Thu Sep 04, 2014 5:21 am
by max
DaemonR wrote:malloc.h is not included in the list of standard headers, nor is the implementation for malloc defined within the document.
What?! What list of standard headers? I hope you are aware that every C library has its own headers that just follow the standards defined in the C standard, to provide an interface that is compliant to the C standard

Re: [SOLVED] Internal Kernel Pseudo-Garbage Collection

Posted: Thu Sep 04, 2014 5:23 am
by Icee
DaemonR wrote:While there is example code using the malloc function, malloc.h is not included in the list of standard headers, nor is the implementation for malloc defined within the document. As I stated before, I've read the standards numerous times, I didn't even need to look at them to know they're not in it.
Wow, you're totally bullet-proof. The malloc.h header is not included in the list because it is not a standard header, while stdlib.h that defines malloc() and friends is.