Julien Jorge's Personal Website

Testing C++ signal-slot libraries

Sun May 24, 2020
en

Did you know that every day several new C++ signal-slot libraries are created? There are so many of them that we estimate there is 1.14 libraries for each C++ developer. Let’s have a look at them.

The puma runs faster than a wardrobe

The signal-slot mechanism is a way to implement the observer pattern. Roughly, the idea is to allow subscription of callback functions to an event: when this event occurs, the functions are called. The signal (or subject) is the one emitting the events, and the slots (or observers) are the callback functions. We will use the term connection to refer to the entity representing the subscription of a slot to a signal.

There is a signal-slot-benchmarks repository on GitHub where somewhere near thirty of such libraries are gathered in a global benchmark. This gives a way to compare them performance-wise, which is a good starting point. I discovered this repository in the announcement of a new signal-slot library on the subreddit about C++. Obviously, as the author of a similar library, I had to enter the benchmark.

And I wasn’t ranked very well.

After a few days of crying in the shower for not even being on the podium I did a lot of measures, reviewed my code several times, and despite that I could not improve the performance very much. There’s something wrong, what do the others do that I don’t? What don’t they do that I do?

It turns out that in my implementation the slots are copied when a signal is triggered in order to avoid handling subtle cases of reentrancy and other modifications of the slot list during the activation. In particular, I want the callback functions subscribed during the execution of a signal to not be called during the current iteration. Nevertheless, a function unsubscribed during the activation must not be called. Long story short: this copy is expensive.

Other implementations don’t worry about this and will simply not accept any modification of the signal during its activation, maybe they will even crash the program. It becomes difficult then to connect a one-shot slot, for example, like a function that unsubscribes itself from the signal when it is called.

This divergence led me to have a closer look at the other implementations and, with the help of the benchmark repository, to compare the libraries not only on their relative performance but also on their functionalities. After all, if one compares the performance of two things, it must be done for the same level of service; otherwise it’s comparing apples to oranges.

Thus I coded some glue code to have a uniform interface to the various libraries in the signal-slot-benchmarks repository so I could apply to them every test I had already implemented in iscool::signals. Obviously, considering the provenance of these tests you can guess that there is a bias toward my implementation. Also, it occurs that I wanted my library to have the same behavior than Boost.Signals2 on several aspects, that’s why these two libraries pass all the tests (or almost, I’ll come back to this).

Nevertheless, even through other libraries do not pass some tests for reasons of diverging implementation choices (i.e. not necessarily bad choices), some of them are still very good tools. Let’s see that in detail.

List of the libraries used in the analysis

Below are the libraries used in the benchmark, the tag I will use to refer to them (often a trigram) and the date of the latest update at the time I’m writing this (April 2020).

You’re all different

There’s a lot of stuff there. First of all we can see that some libraries are not updated since a long time. Then, if we look in detail we’ll see some subtleties, for example:

cps, css, jls and others require that every callback function connected to a signal must be a method of an instance deriving from a given class. To use object-oriented programming vocabulary, imagine that one must derive from Observer in order to subscribe to a Subject. It’s absurdly painful with this design to connect a free function or a lambda.

aco, jos, nls, nss, psg, wnk, yas and maybe others need a reference to the signal in order to cut a connection. Other implementations allow to keep a connection after the destruction of the corresponding signal (they are then automatically cut) and to assign new connections to existing instances.

aco, css, jos, mws, nls, psg, pss, sss and maybe others require that the slots do not return anything. Other implementations allow the function to return a value and sometimes also provided different methods to aggregate their results.

bs2, ics, jls, lfs, lss, mws, nes, nod, nss, psg, vdk and wnk have a method to tell if a signal has at least one connected slot. The other libraries have no such method.

Only asg, bs2, dob, evl, ics, jls, ksc, lfs, mws, nod, pss and vdk can tell if a connection is active or not.

aco, bs2, cls, ics, jls, ksc, lfs, mws, nod, nss, psg, pss, vdk and yas have a method to disconnect all the connected slots at once. asg, cps, dob, evl, jos, nes, nls, sss and wnk do not have such method but also do not forbid the assignment of an empty signal to an existing instance (s = signal()), supposedly for a similar result. css and lss have no method to disconnect all the slots and also forbid the assignment.

css, lss, mws, nod, nss and vdk disallow swapping two signals.

cls, dob, ksc, lfs and vdk permit to access a signal from several threads. aco, asg, evl, ics, jls, jos, lss, mws, nes, psg, sss, wnk and yas are exclusively mono-thread. bs2, cps, nls, nod, nss and pss provide both configurations.

Finally, there are also several small singularities like having a different connection type for each signal type (jos), having as many signal types as there are arities for the slots (signal0, signal1… in psg), returning a private type from the signal class when registering a slot (nls, it’s hard to store the connection in this situation) and other things I would describe as… let’s say unexpected.

The writer’s view

When I started working on iscool::signals I had three main goals, all resulting from the experience acquired with the game I was developing:

  • single thread: I had barely ever needed to share a signal with several threads. Single threading is my default case, multi threading is the exception. I would rather make an effort to handle the synchronization for the few cases where I need to share the signal rather than to pay the unneeded synchronization in the general case.
  • Boost.Signals2-like: we had initially used Boost.Signals2 as the first implementation in our game. Well-known and well-tested, this library was the best choice to start with. A requirement of the replacement was thus to have a compatible interface to make the transition seamless.
  • compilation duration: of the major problem we had with Boost was that its use had a big impact in the compilation time, mostly due to the many headers and templates to parse and to the symbol duplication in the object files (say welcome to extern template!).

Consequently several of the libraries listed above are in my opinion clearly unfit. If your library is intrusive, or does not allow to connect a lambda to the signal, or if its implementation is mostly in headers, then I won’t be able to use it.

What are you doing?

Back to the tests. In order to compare the libraries I took the tests from iscool::signals then added a few more. The results are listed in the signal-slot-benchmark repository.

The tests are listed in four categories:

  1. activation: what happens when a signal is triggered,
  2. argument: how the signal passes its arguments to the callbacks,
  3. connection management: do I have to keep the connections and will the disconnection prevent the call,
  4. swap: what happens if I swap two signals.

Before taking a tour of the results, a few words about how to interpret them. The easiest case is obviously the one where the test passes. In this case, there’s nothing more to say. When it doesn’t, it may simply be the consequence of diverging design decisions between iscool::signals and the tested library, in which case one cannot actually label the library as deficient. For example, iscool::signals guarantees that the callbacks will be called in the order they are registered. This is only a design choice (on which we relied in our games) and certainly not an inherent quality of a signal-slot system.

When the tested library does not provide the necessary methods for the test, the result is simply ignored. After all, if the function is not available it is not erroneous nonetheless.

Finally, there are libraries that crash some tests straight. There it becomes clearly a problem, even more because the tested cases seem legitimate. For example, disconnecting a slot during its execution is a situation I encounter frequently in the real world and unfortunately some libraries do not handle it.

Signal activation

Let’s begin with the activation. The first test is the most obvious: will the activation of a signal triggers the call to a function connected to it? All the libraries pass this test.

Second test, are the callbacks called in the order they are registered? Only cls, cps and nss_ts do not validate this test. As explained before, this is more a design choice than a flaw.

Third test, if I connect a function during the execution of a signal, will this function not be called in the current execution loop? asg, css, jls, lss, mws, nss_tss, sss and vdk do not pass this test. Moreover, aco, cls, cps, evl, nes, nss_ts, pss_st, wnk and yas crash the program during this test. Once again, the decision to execute or skip the callback during the activation of the signal may be a design choice. I personally consider that a function not registered at the time when an event is initially emitted should not be notified of it, but after all your mileage may vary. In any case, there is no reason to crash in this situation.

Fourth test, if I disconnect a slot during the execution of a signal, will the associated function not be called during the current execution loop? aco, cps_st, nod, nod_st, nss_st, nss_sts, nss_tss and psg do not pass the test. cls, cps, nss_ts and yas crash the program during the test. For this use case I cannot imagine a situation where it could be accepted for a disconnected function to still be called.

Fifth test, can I trigger a signal while it is running? In other words, can a signal be activated recursively? All implementations pass the test except cls, cps and nss_ts who crash the app.

Finally, the sixth and last test in this section, will a function registered twice in the same signal be actually called twice when the signal is activated? The way I have implemented the interfaces to the various libraries does not allow to register the same function twice for the implementations that require the slot to inherit from a specific class, consequently aco, cps, cps_st, css, jls, nes, nss_st, nss_sts, nss_ts, nss_tss, psg, sss, vdk, wnk and yas are not tested. All the other libraries pass the test.

Overall, only bs2, bs2_st, dob, ics, ksc, nls, nls_st and pss pass all the tests in this category.

Activation with an argument

The first test in this category checks that the signal accepts an argument and actually passes it to the final callback. All implementations pass this test.

The second test ensures that the argument is never copied when both the slot and the signal take their parameters by address. Once again, all the tested libraries pass this test.

Finally, the third test checks that the copy count of the argument is minimal when the signal and the slot take their parameter by value. In practice, I had never found an implementation of a function wrapper that does not create at least one copy when the arguments are declared by value. Consequently, I’ve set the threshold to pass the test at one copy at most. lfs is the only implementation that passes this test.

Update 2020–05–26: Following a discussion on Reddit where u/dag0me pointed that some implementations call std::forward in a loop to pass the signal argument to the callbacks, I have added a test that checks if a temporary value passed as the argument is actually passed to every callback. We expect that by calling std::forward in the loop then the argument will be moved to the first callback then be empty for the next, thus failing the test. cls, cps, cps_st, dob, ics, ksc, nes, nls, nls_st, nss_st, nss_sts, nss_ts, nss_tss, wnk and yas fail the test.

Overall, only lfs passes all the tests in this category.

Connection management

Once again we’ll begin with a very simple test: does a signal with no connection actually tells that it has no connection? Obviously all the libraries with a method to test if a signal is empty or not pass this test, i.e. bs2, bs2_st, ics, jls, lfs, lss, mws, nes, nod, nod_st, nss_st, nss_sts, nss_ts, nss_tss, psg, vdk and wnk.

The second test is symmetrical to the first one, does a signal with a connection actually tells that it has a connection? The same libraries validate this test.

Third test, will the function connected to the signal be called when I trigger the latter, even if I don’t store the object representing the connection? aco, cls, cps, cps_st, css, evl, jls, nss_st, nss_sts, nss_ts, nss_tss and sss do not pass the test. I personally have no definite opinion on disconnecting the slot when the connection goes out of scope (or, in other words, imposing the storage of the connection to keep it active). On one hand storing them force the programmer to pay attention to the callbacks’ lifetime. On the other hand allowing to ignore the connection object reduces the noise when one is sure that the signal will be destroyed before the dependencies of the connected functions.

Fourth test, will a registered callback not be called if the signal is triggered after the connection has been cut? All libraries pass this test.

Fifth and last test, will the function not be called if the signal is reset? As a reminder, some implementations do not provide any method to reset the signal, and for some of them it is possible to assign a freshly created signal to an existing instance in an attempt to clear it. For this test, css and lss have no method whatsoever to reset the signal. cps, cps_st, dob and mws do not pass the test, while sss crashes. The other implementations pass this test.

Overall, only bs2, bs2_st, ics, lfs, nes, nod, nod_st, psg, vdk and wnk pass all the tests in this category.

Swapping

Swapping a pair of signals is a feature I use in order to have signals that automatically disconnect their slots when triggered. In such case, when I trigger the signal I proceed in three steps:

  1. create a temporary signal,
  2. swap the temporary signal with the active instance,
  3. trigger the temporary signal.

At the end of the third step all callbacks are disconnected. Eventually new connections may be created during the activation, in which case they will be triggered at the next iteration.

Several libraries disallow the swap of two signals. css, lss, mws, nod, nod_st, nss_st, nss_sts, nss_ts, nss_tss and vdk are in this situation. Consequently they do not pass the tests below.

The first test swaps two signals without connection then checks that they are both empty. All implementations pass this test except for cls who crashes.

The second and third test swap an empty signal with a signal having respectively one and two connections, then the signals are triggered. cps, cps_st, dob, nls and nls_stdo not pass this test. cls and sss crash the program.

The fourth test swaps a signal having a single connection with another signal having a single connection too, then they are triggered. cls, cps, cps_st, dob, nls and nls_st do not pass the test. jls and sss crash.

The fifth test swaps a signal having a connection with a signal having two connections. The sixth test swaps a signal having two connections with a signal having two connections too. Then the signals are triggered. The results are the same than for the fourth test.

The seventh test swaps a signal with another one during the activation of the former. Only bs2, bs2_st, evl, ics, ksc, lfs, pss, pss_st and yas pass the test. cls, cps, jos, psg, sss and wnk crash in this situation.

Finally the last test checks that the connections of swapped signals are well associated with the instance having received the signal from which they were initially created. cls, cps, cps_st, dob, evl, lfs, nls and nls_st do not pass the test. jls and sss crash.

Overall, only bs2, bs2_st, ics, ksc, pss, pss_st and yas pass all the tests in this category.

Merging all of this

When I began the migration of the tests from iscool::signals to apply them to the other implementations, I was convinced that the results would mostly feel like a conformance test to Boost.Signals2 instead of a set of expected features for any library of this kind. Nevertheless I am still astonished by the disparities in the results.

Some properties feels optional, like the order in which the connected functions are called by the signal, for example, but others feels essential to me, like adding and removing connections from a signal while it is activated. I can’t keep count of the situations in our games where the first thing a callback does is cutting the connection to the signal that has called it. The same goes for swapping signals, which is a feature we used several times.

Finally there are the crashes. After all, whether the signal does or doesn’t support a feature like swapping or something else, why not, but in this case the implementation must prevent it. Kudos to the authors who took care to prevent swapping when their implementation could not handle it.

To come back to the performance of iscool::signals regarding the other libraries — since it was the problem that started all this analysis — it occurs that for equivalent functionalities it is still the most efficient implementation. Hooray! Besides, until I add the test on the copy count of the arguments by value, it was the only implementation with Boost.Signals2 that passed all the tests. Why the heck did I add this test? Just while I was closing this article… Whatever, it will give me another thing to improve.

To conclude, if you have also coded a signal-slot library, I can only encourage you to add it to the benchmark and the tests. It is both educational and rewarding, and it will help you to see how well you do regarding to the existing libraries.