core: fix race condition in DefaultCallsite::register

[hds]

Motivation

There were two separate race conditions related to registration of
callsites. In both cases, it was possible that event or new_span
could be called before register_callsite had finished executing for
all subscribers.

The first case could be invoked when multiple (thread local) subscribers
were registering the same callsite and could cause some subscribers to
not receive a call to register_callsite at all. This case was fixed
in #2938.

The second case could be invoked when multiple threads reach the same
event or span for the first time and can occur in the presence of only a
single global default subscriber. The subscriber may receive calls to
event or new_span before the call to register_callsite has
finished executing. This may occur even with a relatively fast
register_callsite implentation - although it is less likely. A slow
implementation is more likely to trigger the error.

Solution

This change fixes the race condition by forcing any calls to
DefaultCallsite::register which run while another thread is
registering the same callsite to wait until registration has completed.

This is achieved with a loop around the check on the atomic representing
the registration state for that callsite. It will hotloop until the
registration is complete.

Tests have been added to both tracing-core and tracing which invoke
this error case and always fail when testing the previous code.

Fixes: #2743

[jstarks]

If we must spin, then at least use std::hint::spin_loop().

[hds]

Thanks for the tip!

This currently causes a deadlock in the case of reentrant traces from inside a subscriber, so I'm trying to come up with a solution that avoids that issue.