Clarification of C++ kernel development issues

[kodyob]

I am looking for clarification of the issues that would be encountered in using C++ to develop a kernel. The common advice is that if you use C++, don't use exceptions and don't use the new keyword for memory allocation in the code.

I am wondering why? I wrote two simple C++ programs. One created an int variable statically and the other using the new keyword. I produced assembly code, and while there were differences, it is not clear what it is in the dynamic version that would cause problems.

If you have any links to articles or suggested reading that might make the problems clearer, or any info. on the C++ runtime, please share.

[eekee]

I also don't understand why `new` is discouraged. What can it do beyond calling malloc()? And a kernel malloc() is recommended. Perhaps it's that it would have to use the kernel-specific malloc(), not the one in the standard library, but I'm just guessing. Incidentally, I'd test it with larger data sizes than int. In most languages, ints and other single-element data types (from char to vlong) get treated specially; they're passed by value where other data types are passed by reference. They may be allocated differently too.

Exceptions I don't know much about, but they do mess with execution flow and maybe the stack. I think you'd have to understand exactly how they work before you use them in your kernel.

[nullplan]

OK, the advice about the new keyword is dumb. You do need to implement operator new yourself, but that isn't actually all that horrible. Once you have a memory allocator running, it should just be

Code: Select all

void *operator new(size_t n) {
  void *p = kmalloc(n);
  if (!p) panic("out of memory!");
  return p;
}
void operator delete(void *p) { kfree(p); }

And here we already get to the second part. The throwing operator new must not return an invalid pointer, and the nothrow version just calls the throwing one. To signal failure, it must throw. But exceptions should be avoided in a kernel environment, right? Well, I happen to disagree. Exceptions are a problem because they require runtime support, and that support is not easy to implement. But once you have it, you should be able to use them normally if you avoid the semantically undefined throwing of exceptions through the outermost layer of the kernel. I mean, what does it mean to be throwing an exception through the system call layer into userspace? Is it even sensible to be throwing anything from an interrupt handler? So, if you do use exceptions, you must include last-ditch exception handlers to the outermost layers of the kernel: The kernel entry point, all of the interrupt and exception handlers, and the system call handler.

Ooh, but exceptions have terrible performance. Yes, but to me, correctness and simplicity far outweigh performance.

[kzinti]

There is no reason not to use the new operator. Any advice telling you not to is... uninformed. I am using new a plenty and have no problems.

I am even working on getting C++ exceptions working, but I wouldn't advice anyone to go that route. It is rather complicated and very toolchain specific. For example, I am using a custom GCC cross-compiler and in this scenario libgcc contains the stack unwinding code. That code relies on pthread to handle multithreading scenarios (which you kernel certainly is, unless you put a big kernel lock around everything).

Now implementing pthread inside your kernel is quite a lot of work and probably not what you want. You can of course provide your own stack unwinding implementation that doesn't rely on pthread, but that's probably even more work than implementing the required pthread functionality.

The main difficulty is around reentrancy: you have to make sure that your C library, your C++ library and your libgcc can all handle being interrupted at any time. What happens if you get an interrupt while a c++ exception is being thrown and your interrupt handler also throws an exception? The libgcc code might be holding some mutex... Does it work? Does it hang? The C library was easy as I am using newlib right now and it has very good support for reentrancy. But libstdc++ and libgcc were not built for that.

Anyhow, I am having a lot of fun getting full standard C++20 working in my kernel... Like nullplan says, it is doable but it is a significant amount of work. I might just end up ditching the exception stuff entirely, we will see...

I am not buying the performance argument either... Correctness and scalability are more important especially with modern SMP architectures.

But back to "new"... It is no different than using malloc() and using it is how you write C++ code. So go for it... Just be aware that without C++ exceptions you don't have a clean way to report a problem from the constructor. But I can imagine a pattern where you set default values in the constructor and use a separate Initialize() method where needed.

[xeyes]

Languages are a spectrum, some don't even bother to use assemblers (they use hex (AKA machine code) editors) while some choose to write device drivers with javascript so they can scale with the web and the cloud.

A cost benefit analyze for yourself and your project is probably the prudent way to go.

Benefit:

If you've never written serious code without tons of templates and abstracted interface classes, you would benefit quite a bit from having all these very useful high level features available to you.

If on the other hand, you come from a lower level background (for example you've worked on complex firmware/embeded OS that does not use any dynamic allocation throughout) then C++ might even confuse you and cause you to implement subtle bugs, rather than making things easier for you.

Might also want to spend some moments thinking about what aspects of your project can make use of the features and how big a help these features might be. This point can easily get very controversial so you'd have to make the call yourself. For example, do you want to panic on any bit of unexpected situation so things never go on with mistakes only to blow up in an ugly mess? Or do you want to spend the resource (time) to implement recovery features so that the 20th level exception handler can usually get everything back on track without the user even noticing anything different?

Cost:

Firstly these high level features aren't magic, at the end of the day they are plain C code (or machine code) themselves that have to be written.

You either design and implement the needed runtime and its dependencies. Which can be a big commitment, it's certainly easy to type "new', but is it so easy to implement efficient and correct allocators?

Or you copy someone else's code. At the same time losing the opportunity to tackle it yourself in a controlled fashion, committing yourself to other's ideas, designs and licenses, as well as risk pushing your whole project closer to "change CONFIG_LOCALVERSION, make menuconfig and then make install".

Secondly, dynamic stuff has an overhead at runtime, bigger memory footprint, more CPU cycles and more power burned running the language runtime. Unless you're 200% sure that your hardware is so fast that indeed javascript would be too fast for your drivers, I'd be cautious rather than too optimistic.

Lastly, there can be many moving pieces and other challenges a "very simple" kernel, it does not help when you additionally have to mentally keep track of "oh this class is a 15th generation descendant of a base so its constructor might need ABC and affect XYZ" while debugging it. Usually, there won't be any fancy high level debuggers like in user space to help you keeping track of these either, unless you'd like to spend a few years developing one before starting on your OS project.

At the end of the day though, for a hobby project, this is a Coke vs Pepsi thing. Namely, if in doubt, don't take "who who who on the internet"'s words for granted, try both and decide on what you like

[kzinti]

(...) to implement recovery features so that the 20th level exception handler can usually get everything back on track(...)

That is not a realistic scenario in any software I have worked on. Neither the depth of exception handlers nor the idea that you can get things back on track and somehow erase what caused the exception.

Exceptions are a way for a function to report an error when there is no context to know what should be done. The caller might know (in which case you could just have returned an error code instead). But if the caller doesn't know, the caller will have to forward the error up the call stack by also returning in. Repeat. Exceptions work the same way, but will take care of cleaning up objects created along the call stack for you.

it does not help when you additionally have to mentally keep track of "oh this class is a 15th generation descendant of a base so its constructor might need ABC and affect XYZ"

Inheritance is bad in general. If you have a hierarchy 15 deep, you don't know what you are doing anyways and you will have problems debugging your code no matter what.

[bzt]

I am looking for clarification of the issues that would be encountered in using C++ to develop a kernel. The common advice is that if you use C++, don't use exceptions and don't use the new keyword for memory allocation in the code.

I am wondering why?

I also don't understand why `new` is discouraged. What can it do beyond calling malloc()?

OK, the advice about the new keyword is dumb. You do need to implement operator new yourself, but that isn't actually all that horrible.

You all misunderstood. When the common advice says don't use exceptions and the new keyword, they say that because on bare metal you simply don't have the language support for them. Nothing provides it, and when you start writing your kernel you don't have the required run-time. Simple as that.

So the required steps are:
1. start without exceptions and new, as the common advice says
2. implement a lot of things in your kernel, including run time support for exceptions and a memory allocator
3. then, and only then, not a bit sooner you can start using C++ language features like exceptions and new.
4. for a full C++ language support, port libstdc++ and librtti (over 20Mb in .so, but the static .a library is bigger than 5Mb too)

Of course adding support for all the C++ features is quite a task. For a good example, take a look at Circle. That is a bare metal library that provides access to RPi peripherals (not relevant now) as well as the required C++ run-time. If you statically link your kernel with Circle, you can start using exceptions and new ASAP, but you need that library which implements them. (I've picked this example because it is independent to the language and the build environment, not part of GNU toolchain etc., and being a standalone library it's easier to study its source. Librtti's and libstdc++'s source together is bigger than your kernel ever going to be).

Cheers,
bzt

[kzinti]

Right now I see this on my machine:

Code: Select all

libc.a  - 8.1 MB
libstdc++.a 8.9 KB

kernel: 1.3 MB

Would my kernel be smaller without C++ code? Of course... less code takes less space.

Whether or not that is a problem is in the eye of the beholder.

But I do agree, adding / patching runtime support in for the kernel is a significant amount of work and not many people have done it. Not for beginners.

[bzt]

On my machine

Code: Select all

$ du -h libc.a libstdc++.a libstdc++.so.6.0.28 
5.3M	libc.a
5.6M	libstdc++.a
20M	libstdc++.so.6.0.28

Would my kernel be smaller without C++ code? Of course... less code takes less space.

It's about the ratio. C++ library takes magnitude times bigger space than my entire kernel... It's even in par with the Linux kernel, which is known to be a monolithic bloated monster:

Code: Select all

$ du -h vmlinuz-linux 
8.6M	vmlinuz-linux

When the language support is bigger than the code itself, that must raise some questions about its reasonability.

Cheers,
bzt

[kzinti]

Oh right, my sizes above are "debug" builds with symbols, asserts and everything in.

You got me curious. I willl dig through the kernel file to see why it is so big. I don't mind the libs being gigantic, but the final kernel image should be stripped out of anything it doesn't need.

[bzt]

Oh right, my sizes above are "debug" builds with symbols, asserts and everything in.

And I provided the sizes of the default files that were shipped with my distro.

You got me curious. I willl dig through the kernel file to see why it is so big. I don't mind the libs being gigantic, but the final kernel image should be stripped out of anything it doesn't need.

Yeah I agree. When I first started to use Linux, the kernel was about 1M if I compiled everything in. For a long time kernels were about 5M in size, but why did my distro installed such a huge kernel (size almost doubled!!!) that's a really good question. The kernel's size in general became such a big problem, that Linux kernel developers are helding meetings and summits about it since 2014, ROTFL! Phoronix has good summaries and keeps an eye on how bloated the Linux kernel really is (hint: very much), but they only consider the source, not the binary.

Cheers,
bzt

[thewrongchristian]

Ooh, but exceptions have terrible performance. Yes, but to me, correctness and simplicity far outweigh performance.

Do they have terrible performance, though?

If your common case is no errors, no exceptions, then exceptions based handling could well be faster, as you'll be spending less time checking return codes and weaving over error handling code.

When you do have an error, then it may be more expensive to unwind the stack and propagate the error. But it may be that the performance improvements from a cleaner common case exception free code path will outweigh the performance hit when things have already gone wrong anyway.

And agreed, correctness and simplicity make the issue moot anyway. A forgotten error code check might make all the performance improvements in the world a total waste of time.

Make it work first, then make it work fast.

[nullplan]

4. for a full C++ language support, port libstdc++ and librtti (over 20Mb in .so, but the static .a library is bigger than 5Mb too)

Who needs full support? We are still building a kernel here, still in "freestanding" environment. I kind of doubt you need a 5MB lib just for RTTI and exceptions. It's just that a freestanding C++ implementation has more requirements than a C implementation.

Do they have terrible performance, though?

Yes. I can't find the stats right now, but if the exception triggers, the latency is orders of magnitude above the other error handling methods. The "zero cost" exception handling model is especially bad about this, since in that case once the exception starts you have to start reading tables. And reading DWARF2 debug information in order to unwind the stack. Error code handling is way better optimized. And the costs of error codes vs. "zero cost" if no exception happens doesn't appear to be all that bad. Not orders of magnitude, certainly.

However, it usually doesn't matter because exceptions should be rare.

[kzinti]

The main thing I am after here is access to all the std c++ library, algorithms and containers. That does mean having to support exceptions.

The way I see it, exceptions will only be used to signal the lack of resources (memory or otherwise, but really probably only memory). So this should indeed be rare. If I do need to reverse course on this, it should be a (relatively) easy refactor.

But really I am doing it because I think it's a fun challenge and nobody else seems to be doing it.

[xeyes]

That is not a realistic scenario in any software I have worked on.

Might want to keep an open mind rather than the stance of "I've only ever had Pepsi so Coke MUST taste REALLY REALLY BAD"

Inheritance is bad in general.

Why don't you have ISO fix the C++ standard for you by removing this 'bad feature'