View on C++ exceptions in kernel space

[Griwes]

There are some general rules for using exceptions.

1) If you don't use RAII, don't use exceptions (because exception handlers will look just like error code handlers).
2) If you want to use exceptions just like error codes, don't use exceptions (don't "catch'em all", one by one).
3) If you want to catch all exceptions in the same function they were thrown, don't use exceptions.
4) If you want to use exceptions as a way to change control flow, don't use exceptions.

Brendan has failed at all of those points; that's why his example should not use exceptions, but would have to be completely rewritten before applying exceptions on them (and the rewritten case was already posted).

[rdos]

Exceptions are much cleaner, more elegant, easy to read, and make everything a lot easier?

A little addition to my previous post: there is a little chance that the sarcasm in this statement was not obvious. Even in my little simple (not realistic, though) example I do not find exceptions any better than the traditional error handling. It resembled the same code flow structure anyway and try-catches look harder to read (for me). What about all the assembly procedures? I do not even know how I make them to throw an exception.

Let alone all the overhead it makes to have them at all. Even if that is not the issue, I do not where I would use exceptions. But that also because I am not expert. Sorry for disturbing this thread.

Part of exception handing is to handle CPU exceptions like page fault and protection fault. Those are the typical errors you expect from assembly code. The error code approach cannot handle arbitrary page faults, and rather would call some different code path which cannot interact with the code and return an error code. The end result would be an abnormal exit to the process (or a kernel panic). With C++ exception handling, you could handle page faults locally as well, or delegate them to some other handler. Only if these exceptions propagate all the way up will the process go into abnormal exit.

[rdos]

According to C++ Exception Support all you need to port is libcxxrt and libgcc_eh, which doesn't seem that massive, also I must redirect you to my first question;

I've still not understood how these things relate, and the wiki article isn't that helpful. I'm pretty sure about how to implement libgcc multithread support, and I will definitely want multithreaded support. I also have identified functions called in order to create the exception support ( __register_frame_info and __deregister_frame_info). The problem for me is how to integrate this with newlib, signals, TLS-support and CPU exceptions.

[Owen]

You can't use the C++ exception handling library for C++ exceptions. The compiler assumes that only function calls (and throw statements, obviously) will throw; if an exception occurs elsewhere, you'll land in whatever the nearest exception landing pad was (if you're lucky).

The LLVM guys have discussed this kind of thing in the past, but generally making the code generated able to take an exception at any point really hampers optimizations and will cause massive exception unwind table bloat. It's not presently something that they support.

__register_frame_info and __deregister_frame_info are called at library/executable load/unload (respectively) to pass the exception handling support code a pointer to the unwind information (which is usually contained in the .eh_frame section). I unfortunately can't be more specific than that; I haven't really looked into them. They're mainly aimed at JIT compilers, as far as I know; though I assume the dynamic loader must invoke them also.

[rdos]

You can't use the C++ exception handling library for C++ exceptions. The compiler assumes that only function calls (and throw statements, obviously) will throw; if an exception occurs elsewhere, you'll land in whatever the nearest exception landing pad was (if you're lucky).

That seems strange to me. I'm most familiar with Win32 exception handling, which does include page fault exceptions and other CPU exceptions being converted to ordinary exceptions. These also interact with the debugger support in Win32.

So if I understand you correctly, there is no such integration in GCC / LLVM?

I forgot to mention that I don't understand how the debugger (gdb) fits into the picture.

It would be helpful with some document that describes this in more detail.

[Owen]

You can't use the C++ exception handling library for C++ exceptions. The compiler assumes that only function calls (and throw statements, obviously) will throw; if an exception occurs elsewhere, you'll land in whatever the nearest exception landing pad was (if you're lucky).

That seems strange to me. I'm most familiar with Win32 exception handling, which does include page fault exceptions and other CPU exceptions being converted to ordinary exceptions. These also interact with the debugger support in Win32.

Win32 exceptions are different beasts. In particular, they're divided into "asynchronous" and "synchronous" exceptions. The Itanium C++ ABI (Which GCC uses most places; ARM is an exception and has a slight variant) only does exceptions which map to Win32 "synchronous" exceptions.

Win32 asynchronous exceptions are raised in response to page faults and such. They don't interact with C++ exception handling at all - in particular, they don't cause destructors to be invoked.

Win32 synchronous exceptions are raised by the RaiseException function, and are used by C++ exception handling (among others - e.g. .net uses them internally also). These do cause destructors to be invoked.

Invoking destructors from arbitrary instructions isn't done because it's rarely needed (most such exceptions can be fixed on the spot e.g. by mapping a page or are immediately fatal) and massively complicates the control flow graph.

So if I understand you correctly, there is no such integration in GCC / LLVM?

Correct, because only Win32 has really had exception handling in this manner. You could quite feasibly add support to a similar level to Windows does to the existing interface, however, but it would require libgcc_eh and compiler changes.

I forgot to mention that I don't understand how the debugger (gdb) fits into the picture.

The debugger will need OS support to read/write the process' memory, hook various events (mainly signals), set breakpoints & watchpoints, etc. Additionally, there is generally an "informal protocol" used to notify the debugger of certain events which don't originate within the OS kernel; for example, whenever a new shared library is loaded (or a JIT compiler emits some code), the debugger would like informing about it so that it can read the debugging information. The way this is done on most platforms (Linux, *BSD, Mac OS X) is by the dynamic linker and JIT compilers calling a "dummy" method with a pointer to whatever data the debugger requires. This dummy method does nothing - its implementation can be as trivial as a "ret" - but the debugger can set a breakpoint on it in order to listen for these events.

The debugger is mostly ignorant of exceptions. gdb won't (by default) break when an exception is thrown; only when it goes unhandled and std::terminate() invokes abort() (the debugger will detect the SIGABRT and break there). Normally by this point the useful context is gone which is unfortunate, though one could probably add a dummy method to trap this in a similar to how exception handling information is passed to it (The unwinder can most of the time establish that terminate() is going to be called before it starts actually unwinding).

When I need to debug a problem at the exception site, I generally ask for it to break whenever an exception gets thrown: "break __cxa_throw".

[rdos]

Win32 exceptions are different beasts. In particular, they're divided into "asynchronous" and "synchronous" exceptions. The Itanium C++ ABI (Which GCC uses most places; ARM is an exception and has a slight variant) only does exceptions which map to Win32 "synchronous" exceptions.

OK, and the asynchronous part is signals in Posix then?

Win32 asynchronous exceptions are raised in response to page faults and such. They don't interact with C++ exception handling at all - in particular, they don't cause destructors to be invoked.

That sounds like a design-flaw.

Win32 synchronous exceptions are raised by the RaiseException function, and are used by C++ exception handling (among others - e.g. .net uses them internally also). These do cause destructors to be invoked.

I haven't used this extensively yet, mostly because the original environment I used couldn't handle these.

Correct, because only Win32 has really had exception handling in this manner. You could quite feasibly add support to a similar level to Windows does to the existing interface, however, but it would require libgcc_eh and compiler changes.

Is libgcc_eh similar to asynchronous or synchronous Win32 exceptions? How is the asynchronous Win32 exceptions implemented in gcc, and what is the required OS support. After all, CPU exceptions originates in the OS, and must be "reflected" to usermode somehow. Maybe it is signals that is the implementation for this?

The debugger is mostly ignorant of exceptions. gdb won't (by default) break when an exception is thrown; only when it goes unhandled and std::terminate() invokes abort() (the debugger will detect the SIGABRT and break there).

At least it must react to single step and break point exceptions. But these would be signals, so gdb would hook these signals then?

Normally by this point the useful context is gone which is unfortunate, though one could probably add a dummy method to trap this in a similar to how exception handling information is passed to it (The unwinder can most of the time establish that terminate() is going to be called before it starts actually unwinding).

That's definitely not acceptable behavior.

[Owen]

Win32 exceptions are different beasts. In particular, they're divided into "asynchronous" and "synchronous" exceptions. The Itanium C++ ABI (Which GCC uses most places; ARM is an exception and has a slight variant) only does exceptions which map to Win32 "synchronous" exceptions.

OK, and the asynchronous part is signals in Posix then?

Right

Win32 asynchronous exceptions are raised in response to page faults and such. They don't interact with C++ exception handling at all - in particular, they don't cause destructors to be invoked.

That sounds like a design-flaw.

It's definitely unfortunate. Note however that handling an asynchronous exception doesn't require stack unwinding (if you return EXCEPTION_CONTINU_EXECUTION then it will resume execution from the instruction which casued it)

Correct, because only Win32 has really had exception handling in this manner. You could quite feasibly add support to a similar level to Windows does to the existing interface, however, but it would require libgcc_eh and compiler changes.

Is libgcc_eh similar to asynchronous or synchronous Win32 exceptions? How is the asynchronous Win32 exceptions implemented in gcc, and what is the required OS support. After all, CPU exceptions originates in the OS, and must be "reflected" to usermode somehow. Maybe it is signals that is the implementation for this?

libgcc_eh is similar to synchronous exceptions. GCC is completely ignorant of Win32 exceptions, and uses the same exception model on Win32 it does on other platforms.

Win32 "structured exception handling" doesn't necessarily require compiler support; applications can in fact use it in terms of the APIs exposed by Kernel32.dll

The debugger is mostly ignorant of exceptions. gdb won't (by default) break when an exception is thrown; only when it goes unhandled and std::terminate() invokes abort() (the debugger will detect the SIGABRT and break there).

At least it must react to single step and break point exceptions. But these would be signals, so gdb would hook these signals then?

Correct. The traditional API used for this is ptrace.

--

This is getting somewhat long and quite tangential. Perhaps this can be split into another topic ("Implementing C++ exceptions", perhaps)

[dozniak]

This is getting somewhat long and quite tangential. Perhaps this can be split into another topic ("Implementing C++ exceptions", perhaps)

I find it the most valuable part of this thread, please, continue!

[dschatz]

I've got multicore-compatible c++ runtime support (including exceptions) in my kernel. It expands the size of the kernel by a few hundred kilobytes which I believe is the only downside. Because of my love for RAII it has been worth it.

People claim that exceptions have poor performance but I believe this is a misunderstanding of how it works. The Itanium ABI explains it well but in a nutshell: a try statement has no execution overhead, so in the case where there is no exception, using c++ exceptions is faster than using error code returns (which must be checked whether there is an error or not). When an exception does occur, there is indeed significant execution overhead compared to returning error codes. What this means is that if exceptions occur infrequently (they are exceptional!) then exceptions are higher performance than return code checking.

[OSwhatever]

I've got multicore-compatible c++ runtime support (including exceptions) in my kernel. It expands the size of the kernel by a few hundred kilobytes which I believe is the only downside. Because of my love for RAII it has been worth it.

It is that, a few hundreds of kilobytes is a deal breaker for me, otherwise I would have considered it. Hundreds of kilobytes doesn't sound much but for kernels it is as they are supposed to be small and not pollute the cache too much. I know that GCC adds this overhead but I would be nice to have pure bare metal exceptions without the library baggage.

[dozniak]

Hundreds of kilobytes doesn't sound much but for kernels it is as they are supposed to be small and not pollute the cache too much.

But these hundreds of kilobytes never reach the cache if you don't hit an exception path.

[Brendan]

Hi,

I've got multicore-compatible c++ runtime support (including exceptions) in my kernel. It expands the size of the kernel by a few hundred kilobytes which I believe is the only downside. Because of my love for RAII it has been worth it.

It is that, a few hundreds of kilobytes is a deal breaker for me, otherwise I would have considered it. Hundreds of kilobytes doesn't sound much but for kernels it is as they are supposed to be small and not pollute the cache too much. I know that GCC adds this overhead but I would be nice to have pure bare metal exceptions without the library baggage.

Most micro-kernels are less than 100 KiB, so adding a few hundred KiB just for exceptions would probably triple the kernel's size. For monolithic kernels it seems different (e.g. a few 100 KiB added to a 5 MiB monolithic kernel doesn't sound like much); but it's mostly the same - there'd be about 100 KiB of frequently used "kernel core" that you want to stay in cache (and in TLB) if possible, with several MiB of less frequently used code (drivers, etc) on top.

For bare metal exceptions, in assembly it'd be easy to pass an "address to return to on error" to each function. For example:

Code: Select all

myFunctionFoo:
    ...
    mov edi,.errorHander
    call myFunctionBar
    ...
    ret

.errorHander:
    ...      ;Free anything allocated before myFunctionBar was called and handle error/s
    ret


myFunctionBar:
    ...
    jc .outOfPizzaError
    ...
    ret

.outOfPizzaError:
    ...
    mov [esp],edi
    ret

This actually wouldn't suck - modern CPUs keep track of return addresses and if the error occurs the "ret" would cause a branch misprediction, but it would avoid the need to check for errors in the "happy case" (and for something like "if(status != OK)" there's a high probability of branch misprediction anyway).

Sadly, you can't do this in C because there's only function pointers and no sane way for a function to change its own return address; and if you attempted to add special support for it you'd end up with a gold-plated turd (where massive amounts of bloat are hidden under syntactical sugar) like C++.


Cheers,

Brendan

[Brendan]

Hi,

Hundreds of kilobytes doesn't sound much but for kernels it is as they are supposed to be small and not pollute the cache too much.

But these hundreds of kilobytes never reach the cache if you don't hit an exception path.

It would be nice if code involved in exception handling could be put in a separate section (e.g. instead of ".text") to avoid polluting things like the TLB. The same applies to normal cache lines - e.g. you don't want a 64-byte cache line that contains 32 bytes of "happy case" and 32 bytes of "exception handling".

To clarify (using the "bare metal" example above to demonstrate):

Code: Select all

    section .text
myFunctionFoo:
    ...
    mov edi,.errorHander
    call myFunctionBar
    ...
    ret

    section .exceptText
.errorHander:
    ...      ;Free anything allocated before myFunctionBar was called and handle error/s
    ret

    section .text
myFunctionBar:
    ...
    jc .outOfPizzaError
    ...
    ret

    section .exceptText
.outOfPizzaError:
    ...
    mov [esp],edi
    ret

In this case the cache line that contains the end of myFunctionFoo's "happy case" is likely to also contain the beginning of myFunctionBar's code; and the "happy case" for both functions is likely to be on the same page (same TLB entry).


Cheers,

Brendan

[Owen]

Hi,

I've got multicore-compatible c++ runtime support (including exceptions) in my kernel. It expands the size of the kernel by a few hundred kilobytes which I believe is the only downside. Because of my love for RAII it has been worth it.

It is that, a few hundreds of kilobytes is a deal breaker for me, otherwise I would have considered it. Hundreds of kilobytes doesn't sound much but for kernels it is as they are supposed to be small and not pollute the cache too much. I know that GCC adds this overhead but I would be nice to have pure bare metal exceptions without the library baggage.

Most micro-kernels are less than 100 KiB, so adding a few hundred KiB just for exceptions would probably triple the kernel's size. For monolithic kernels it seems different (e.g. a few 100 KiB added to a 5 MiB monolithic kernel doesn't sound like much); but it's mostly the same - there'd be about 100 KiB of frequently used "kernel core" that you want to stay in cache (and in TLB) if possible, with several MiB of less frequently used code (drivers, etc) on top.

My kernel binary currently sits at 112kB of Thumb-2 code, of which ~20kB is the unwinder (for reference, ~30kB is the heap). This isn't completely onerous.

Also, while unfortunately neither LLVM nor GCC put the code executed on the unwinding path in a separate section (though there is no reason they couldn't be augmented to do so), the majority of the time no machine code will be generated for the unwind path (the unwind - including destructor calls - is completely expressed in the .eh_frame sections. In general only catch() blocks require machine code emission.

Also, rdos, I make a correction to my earlier statement: GCC can apparently, via the -fasynchronous-unwind-table option, generate exception data which is instruction precise (which does support exceptions for CPU exceptions). I'm not sure if this is any more featureful than Windows' asynchronous unwind support, however