pthread condition variables

[rianquinn]

I am currently writing a hypervisor in C++, and we are using libc++.
https://github.com/Bareflank/hypervisor

I got rid of the "-fno-threadsafe-statics" and added my own implementation of pthread_cond_wait and pthread_cond_broadcast (which is what libcxxabi uses). We don't have threads in the hypervisor, but we do have multiple CPUs so in a way, we always have 1 thread per CPU (i.e. no scheduling, but things like a mutex are still needed). Because we don't have the ability to schedule a thread, a mutex and a condition are really just spin_locks.

Implementing a mutex with a spin_lock seems pretty straight forward, but implementing pthread_cond_wait was a little more complicated. I was wondering if you guys wouldn't mind taking a look at what I have done, and try to poke holes:
https://github.com/Bareflank/libbfc/blo ... d.cpp#L249

I do no intend to implement pthread_cond_signal... just pthread_cond_broadcast (i.e. I don't think I need a Semaphore) so I set the condition to 1, unlock the mutex, and wait for a broadcast which sets condition back to 0, and then I lock again. My major concern are race conditions with this implementation. How is this usually implemented? Does anyone have any decent examples on how I might go about this if I have done it wrong?

Thanks in advance,
- Rian

[Boris]

Pthread_cond_wait ensures that the thread will sleep while the mutex is released.
Which means, if the condition is protected by the mutex, you cannot set it before the thread sleeps.

the nightmare that will come to you is that you may later wake the CPU ( because the condition triggers) while you are putting in to sleep; therefore you may end up with a zombie cpu if the sleep is scheduled after.

[simeonz]

There appear to be multiple issues with the implementation. (I am not sure that they concern the use case of safe statics, but they break the contract.)

1. pthread_cond_broadcast right now performs what pthread_cond_signal should perform - it signals one waiter. Namely, the waiter that first performs __sync_lock_test_and_set in its busy loop.
2. pthread_cond_wait checks the state in a loop, which means that broadcasting with a flag will never work ("pulsing" the flag can be missed, once issue 1 is solved)
3. As Boris mentioned, there in the interval between pthread_mutex_unlock and the loop check in pthread_cond_wait, signals can be missed

For this limited case (with the interface you require), you could use a counter in place of the flag, and increment it atomically in pthread_cond_broadcast. pthread_cond_wait has to read the counter on entry before pthread_mutex_unlock and poll for increment to the next value in the loop. Memory ordering when accessing the counter is important as usual.

Edit: Above I said that the waiter needs to "poll for the next value", but actually the poll needs to be for a change (any increment.) Also, note that wrap around is possible in principle, but for the thread guards case and with enough bits in the counter, it will never happen.