Multithreaded executor behavior

[HarunTeper]

Hello,

while working on the multithreaded executor, I encountered a problem.

Take the following configuration:
One node, with two timers, each timer with a period of 1000ms, and an execution time of 1000ms per timer. Both timers are in a mutually exclusive callback group.
Due to the current structure of the multithreaded executor, only one timer will be executed, while the other one will never be executed.
I suppose this is due to one thread updating the wait set, while the other one is blocked by the execution. By the time the thread that occupies the callback group is finished, the task of the just finished timer is ready again, and the second timer task is never executed.

In my changes, I updated the behavior of the multithreaded executor, so that the wait set is only updated once all the work that is available is done. This guarantees that all callbacks in a mutually exclusive callback group will be executed, even if some of them are ready again. This change should align the behavior of the multithreaded executor more with the behavior of the single threaded executor.

If you have any suggestions or feedback for this idea, please let me know.

[sloretz]

@mjcarroll assigning you to review 🧇

[jmachowinski]

Hm, we ran into the same issue, but as this is a FIFO scheduler, this is intended design from my point of view.
I would say the bug is more on the side, of your timer taking to long to compute.
The system should output a fat big warning in a case of a timer overrun though.

[fujitatomoya]

if the scheduler dispatches the same thread id in this while loop (or if specific thread id is not just dispatched once yet), and if there are no entities available in this while loop, this while loop will keep spinning indefinitely with some threads. i believe this will eat up the CPU resource and waste.

besides that, i do not think this can fix the original problem stated in the report. if the callback group is mutually exclusive, it just cannot execute the entities under the same callback group.