condition_variable workaround for wait_until with system time change

Question

I have a timer class which uses the std::condition_variable wait_until (I have also tried wait_for). I am using the std::chrono::steady_clock time to wait until a specific time in the future.

This is meant to be monotonic, but there has been a long standing issue with this where this actually uses the system clock and fails to work correctly when the system time is changed.

It has been fixed in libc as suggested here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=41861.

the issue is that this is still pretty new ~2019 and only available in gcc version 10. I have some cross compilers that are only up to gcc version ~8.

I am wandering if there is a way to get this fix into my versions (I have quite a few cross compilers) of gcc? - but this might prove difficult to maintain if I have to re-build the cross compilers each time I update them or such.

So a better question might be, what is a solution to this issue until I can get all my tools up to gcc v10? - how can I make my timer resistant to system time changes?

updated notes

• Rustyx mentions the version of glibc needed is 2.3.0+ (more for my ref - use ldd --version to check that)
• glibc change log showing the relevant entry for the fix supplied by Daniel Langr: https://github.com/gcc-mirror/gcc/blob/master/libstdc%2B%2B-v3/ChangeLog-2019#L2093
• The required glibc patch (supplied by rustyx): https://gcc.gnu.org/git/?p=gcc.git&a=commit;h=ad4d1d21ad5c515ba90355d13b14cbb74262edd2

Nahshu

I believe this is a matter of libstdc++, not a compiler itself. Couldn't you upgrade your "system" libstdc++ version independently of GCC? Here is a link to changelog. – Justiciable

You need glibc 2.3.0+ (pthread_cond_clockwait) and patch your libstdc++ with the supplied patch – Fara

@DanielLangr I might be wrong here, but isn't libc++ part of the package of the build tools - I think that comes with gcc? - in the link it says its fixed (the very last line of the page) in gcc v10. Also changing libc++ to some other version could break lots of things? - I have not really tried this, other then with a very minor change (like from libc++ v6.0.1 to v6.0.2 - not real numbers!) – Nahshu

@Fara ah, does this mean re-building my libstdc++ from the source? - I assume that is the case. The patch does not look too scary :o – Nahshu

@Nahshu I think you can always build your own version of libstdc++ and tell GCC to use it instead of the system default version / version shipped with GCC. – Justiciable

@DanielLangr so I was just taking a look at my target setup, we are using bitbake to create the rootfs (whcih contains glibc and such), so I think I just need to patch the version of glibc inside my bitbake package since it contains all the source and I only really need it working on the targets. Its probably harder to rebuild gllibc on my native linux distro because I am not setup to do that :o - thanks both, I think this gives me the right direction to explore. Please feel free to update as answers – Nahshu

Answer 1

1. Create a data structure that contains a list of condition variables each with a use count protected by a mutex.

2. When a thread is about to block on a condition variable, first acquire the mutex and add the condition variable to the list (or bump its use count if it's already on the list).

3. When done blocking on the condition variable, have the thread again acquire the mutex that protects the list and decrement the use count of the condition variable it was blocked on. Remove the condition variable from the list if its use count drops to zero.

4. Have a dedicated thread to watch the system clock. If it detects a clock jump, acquire the mutex that protects the list of condition variables and broadcast every condition variable.

That's it. That solves the problem.

If necessary, you can also add a boolean to each entry in the table and set it to false when the entry is added. If the clock watcher thread hast broadcast the condition variable, have it set the bool to true so the woken threads will know why they were woken.

If you wish, you can just add the condition variable to the list when it's created and remove it from the list when it's destroyed. This will result in broadcasting condition variables no threads are blocked on if the clock jumps, but that's harmless.

Here are some implementation suggestions:

Use a dedicated thread to watch the clock. An easy thing to look at is the offset between wall time and the system's uptime clock.

One simple thing to do is to keep a count of the number of time jumps observed and increment it each time you sense a time jump. When you wait for a condition, you can use the following logic:

1. Note the number of time jumps.
2. Block on the condition.
3. When you wake up, recheck the condition.
4. If the condition isn't satisfied, check the number of time jumps.
5. If the count from 1 and 4 mismatch, handle it as a time jump wakeup.

You can wrap this all up so that there's no ugliness in the calling code. It just becomes another possible return value from your version of wait_for.

Materially

Out of curiosity by which algorithm do you plan to detect a jump in the clock ? – Aubergine

I did have a rough backup plan to have a static member variable with pointers to all the instances along with a static function to wake all the threads. That should work - but it requires outside input I wanted to avoid that - but its a reasonable workaround : ) – Nahshu

@Aubergine for me this is not a problem I know when I am going to change the time (based on GPS updates), so calling the function to wake the threads is not a hassle - but otherwise that would be another bunch of work and by the sound of it difficult to do?! – Nahshu

@Aubergine I wasn't concerned about jumps of less than a second. So my code checks the clock every second. If it's not within one second of where it's expected, we assume there's a jump. You can also use the offset between the uptime clock and the wall clock. – Materially

@DavidSchwartz one issue with this is that, for time going backwards its fine. I wake my timers, they recalc the remaining time and off they go - nice and monotonic. The issue is when time is moved forwards, this triggers the condition_variable wait before I have any chance to do anything. Not the end of the world, but it means that I can't get the timer strictly monotonic this way. But its a half decent work around, I at least wont miss any timer events, I will, at worst get extra timer events – Nahshu

@Nahshu When the condition variable is triggered because of a time jump, your code can do whatever it wants in response to that. – Materially

@DavidSchwartz that is true - but then my code, at this point, has no way to distinguish if this is a legit timeout or a time-jump timeout... at this point I am in the situation darune is suggesting where I have to be able to figure out what is going on with the clock - that's a whole pile of effort! - I think I rather patch glibc at that point :) – Nahshu

@DavidSchwartz btw not sure who down voted you - as I say this is a decent work around, but is not quite 100% solution, def worthy of a +1... – Nahshu

@Nahshu Checking what's going on with the clock is not difficult. Just create a thread whose sole purpose is to watch the clock. As often as required, the thread wakes up and checks the offset between wall time and uptime. If it has jumped, you record the jump someplace that other threads can easily find it. If you're using a modern version of C++, you can easily encapsulate all this so that the code is clean everywhere but in this implementation. – Materially

@DavidSchwartz Ah, that's a good point. I guess we are talking about check the time elapsed (like std::chrono::now().time_since_epoch()) vs the system time. I can just get my timer check has_time_jumped() or better still, calc_next_time(int interval_ms) and let some other class figure out the new interval required... I'll give that a go - thanks. If you add that to your answer I'll mark it up :) – Nahshu