Shared libraries and thread/process affinity?

[Artlav]

Good evening.

How does a shared library handle the fact of many threads and processes are accessing it at one?

The full story, including how i understood a concept of a shared library:
A shared library implement some functions that the applications use. It is located in a common address space and is accessed by calling the required subroutines (with addresses provided to application by the loader/linker). It also have some internal state, like tracking which processes use which of whatever resource it provides.

Now, there is a problem - what would happen if there is a bug or an exception in the library? The kernel would not know that, sending the signal to the process that called the lib, triggering it's response or death and leaving the lib in inconsistent state and/or unable to handle exceptions.

So, how is this usually handled?
I tried to direct the calls thru a linker-made gates, that would track the change of affinity, but that had cumulative effects of breaking other things, which had to be fixed, etc, etc...
Eventually everything worked out perfectly, but stepping back i see an intricate (but rugged) Rube Goldberg machine, which gets me here with a question - how is it handled properly/usually?

[Artlav]

Properly designed shared libraries allocate resources local to threads/processes that use them (or use no shared state at all). This solves the problem in its entirety.

I don't see how.
There is exception in the lib, how is it handled?

[rdos]

It is not the shared library that handle exceptions, it is the application. Shared libraries are just code (and perhaps some private data). If an exception happens, and it remains unhandled, it is the application that gets terminated, not the shared library.

[Artlav]

So, shared library can't have any private data, critical sections, and so on?

[Artlav]

Then i'm a bit lost again.
Let's go to mature OS, both windows and linux would do.
There in a dynamic library i can do anything, for example a graphics engine for a game - using OS resources and other libraries.

Most importantly - i can have try except in there, and it would catch exceptions to the library exceptoin handler.
How is that part implemented?

[Artlav]

Widening up the topic, how is exception handling done?

I have the applications give the kernel the signal handler location at start (and re-give after every call of it), that is getting called when exceptions happen.
Thus, if there is an exception in a library, the caller handler is called and chaos ensues.

What is wrong in this way?

[Artlav]

Ok, then question A - how are exceptions normally caught, on OS level?
And question B - how to determine if they are caught in a lib or in the process?

[Artlav]

Depends on what you understand under "exceptions" - is that language exceptions, structured exceptions (like windows SEH) or something else?

Let's say language level exceptions.
There is an error in try block - access violation, division by zero, etc. CPU throws an exception, kernel does something and the application's language RTL gets the control at exception handling entry point, evaluates the situation and jumps to except block with proper variables set.

In my case, the kernel pushes the info about what happened into the process stack and redirects it to the address that was provided by it (usually at process start).
The address is then forgotten (to avoid double exceptions).
At launch, the language level RTL sets this address to the entry point of the exception handling routine.
That re-sets the kernel address back and goes on to the handling part.

Now, the library and the process have their own copies of that (or different) RTL, separate from each other.
If the lib have an exception in try except, then the process would get the call, and the library gets nothing.
The process is killed, the lib is potentially left in inconsistent state.

This way the libs can't have exception handling. But in major OSes they can.
How is that done?

[Artlav]

First of all, language level exceptions are different from access violation or division by zero - these are signals.
Second, the exception handling works by adding an exception handler into some sort of exception handling block that is referenced from stack (e.g. some function scope), which obviously works as you call into the shared library too.

Then i'm mixing terms a little, sorry for that. When access violation happens in a DLL under Windows, the except block is being executed. So, somehow that kind of errors are also handled.

What you described here sounds like exception handling inside the language RTL - each function have it's handler pushed into exception stack, and the main handler then pops the last one and jumps there.

But there is no common RTL between a lib and a process!
The process called the lib, the lib excepts, the process gets the call.

So, does the common libc do the exception stack, and everything there is is centered in it?
Then how do other languages work, an how does it work on Windows?

[Artlav]

Thanks, i'll get some more research then.

[Owen]

The first bit of confusion you seem to express is the "process affinity" of shared libraries. The best way of thinking of them is just as blobs of code mapped into the calling process by the loader. Any inter-process resource sharing is an implementation detail; the code pages may be shared, but the data pages will not be (Or you might just start with them all shared, but marked copy on write, so whenever one of them is written the process gets a private copy of it).

Threads are exclusively associated with processes - and there is no distinction between library code and process code.

Now, for exception handling: This is extremely system and language specific. Windows uses two methods; a legacy method in Win32, and a newer method on Win64. Most other OSes use a method derived from the Itanium C++ ABI, which I will describe here:

1. When a process wants to raise an exception, it calls _Unwind_RaiseException with a pointer to an exception object to be thrown. I have forgotten exactly how these objetcs are allocated, though it must be noted that the exception handling framework includes a method for allocating space to store a certain number of exceptions as a last resort method if you can't get memory from the system allocator (E.G. this method is most likely used to raise std::bad_alloc when out of memory)
2. _Unwind_Throw looks at its return address to find the calling function
3. The object (Shared library or executable) which contains the function is looked up. The .eh_frame section of this shared library contains an ordered list of all the functions in the object for which exception handling is necessary. The table also contains a pointer to a personality routine. Each language defines a personality routine; you are probably familiar with __gxx_personality_v0 if you have done any work with G++.
4. The personality routine is invoked. The personality routine looks at some language specific data (How this is associated with the function is defined by the library ABI) and decides if it needs to handle the exception or if the unwinding should continue. If there is a handler, then the exception handling framework invokes it. The handler can then specify if it has finished handling the exception or wishes to continue processing it. If there isn't a handler, the unwinder follows some more instructions (in the .eh_frame section) to pop the frame and invoke any destructors in it
5. The frame has been popped. The IP of the next frame up is looked up. Jump back to step 3

The EH frame section contains

• Records for locating the information for each function
• Information describing the frames contained within each function
• Instructions describing where to find certain information in a frame
• Instructions describing what native code helper functions to invoke

This is just a general overview. The official specification can be found at the C++ ABI site. This doesn't contain a lot of the information you need to implement it (e.g. the format of the .eh_frame section). A lot of information can be gleaned from the latest version of the DWARF specification; what isn't specified in there is probably only located in the convoluted code of the GCC libsubc++ project. Some of this is also implemented in LLVM's Compiler-RT library; the latter is much easier to follow, but incomplete.

The exception handling ABI documentation is shambolic.

[Artlav]

The best way of thinking of them is just as blobs of code mapped into the calling process by the loader.

Exactly what i have.

There are two problems, however - resource allocation and exception handling.

Let's take two processes, A and B and a library L.

Code: Select all

  /-L-\
 / RTL \
 |      |
 A      B
RTL    RTL

Each have their own language RTL and with it's ways of handling exceptions and getting resources, in particular, memory.

Case 1, memory allocation:
L allocates memory from the kernel for it's heap. Done by the language RTL of it (for whatever reason from straight malloc to object creation and dynamic arrays).
This happens when the L routine is called from A, for example.
The memory allocated is registered to A.
Now, A terminates.
Memory gets released and is later allocated to something else.
But L still consider it occupied, so when B gives a call chaos ensues.

Case2, exception handling:
Once again, we're in L by grace of A.
If L raises exceptions by language means, all is fine.
But if an exception happens on CPU, then the kernel gets it, and calls the exception handler on A.
Exception handler there starts doing it things, but knowing nothing of L it can't conceivably get it into consistent state and into the corresponding except{} block over there.

These are the problems i observed.
How are they solved or avoided?

My approach was to link the lib through a piece of code that pushes and pops the current affinity, so that memory allocated in the lib would be registered to the lib, as well as exception handlers.

When a process wants to raise an exception, it calls _Unwind_RaiseException

The part i still can't find or get at, neither in Windows SEH specs berkus provided, nor in ABI overview, is what happens when exception is not raised by the process, but just happens.

What is the route from kernel to process's handling?

[Owen]

The best way of thinking of them is just as blobs of code mapped into the calling process by the loader.

Exactly what i have.

There are two problems, however - resource allocation and exception handling.

Let's take two processes, A and B and a library L.

Code: Select all

  /-L-\
 / RTL \
 |      |
 A      B
RTL    RTL

Each have their own language RTL and with it's ways of handling exceptions and getting resources, in particular, memory.

Case 1, memory allocation:
L allocates memory from the kernel for it's heap. Done by the language RTL of it (for whatever reason from straight malloc to object creation and dynamic arrays).
This happens when the L routine is called from A, for example.
The memory allocated is registered to A.
Now, A terminates.
Memory gets released and is later allocated to something else.
But L still consider it occupied, so when B gives a call chaos ensues.

If A and B are different applications, then they are in different address spaces. Any memory allocated by L in A is completely inaccessible to L in B. If it wants to allocate memory accessible by both, then it has to do something more complex (i.e. explicitly allocate shared memory

The per-language runtime libraries should be shared libraries, so all libraries and objects in a process which use on language RTL get the same instance. These should probably call down to the 'OS RTL'. Negotiating how resources are released is the responsibility of the application code - i.e. it should be in the library API contracts. For freeing objects allocated by a C++ library, it might be a simple "delete object;". For freeing objects allocated by a C library, it might be "libname_delete(object);".

Case2, exception handling:
Once again, we're in L by grace of A.
If L raises exceptions by language means, all is fine.
But if an exception happens on CPU, then the kernel gets it, and calls the exception handler on A.
Exception handler there starts doing it things, but knowing nothing of L it can't conceivably get it into consistent state and into the corresponding except{} block over there.

These are the problems i observed.
How are they solved or avoided?

My approach was to link the lib through a piece of code that pushes and pops the current affinity, so that memory allocated in the lib would be registered to the lib, as well as exception handlers.

When a process wants to raise an exception, it calls _Unwind_RaiseException

The part i still can't find or get at, neither in Windows SEH specs berkus provided, nor in ABI overview, is what happens when exception is not raised by the process, but just happens.

What is the route from kernel to process's handling?

On POSIX platforms, processor exceptions are propogated by signals. If I wanted to raise language exceptions for processor exceptions, I would have the process' signal handler throw the exception. Note that many platforms (e.g. Linux) don't allow exceptions to be thrown from signal handlers.

Windows raises exceptions from within the kernel, but Win32 SEH is kernel managed. Win 32 SEH works by establishing a linked list of "exception handler blocks" on the stack. The unwinder moves unwinds to each block in turn, until the exception is handled or the process dies. Raising an exception under SEH is much faster than Itanium C++ ABI EH, however, establishing the exception handler blocks adds some overhead to the program. C++ ABI EH is therefore generally regarded as preferable, as exceptions should be rare.