Every once in a couple hundred runs of a downstream fuzzer test, we see
a fuzzing test freeze while waiting for a thread to be suspended. The
main thread is frozen because it's waiting to suspend either the alarm
or rss thread which is stuck waiting for an exception they sent out to
be handled. Specifically, both threads send out a synthetic
`ZX_EXCP_THREAD_STARTING` exception to be handled by the crash handling
thread which sets up an exception channel on the whole process with
`ZX_EXCEPTION_CHANNEL_DEBUGGER`. This is the only channel type that
listens to thread stop/start exceptions. Normally, the exception would
be ignored and the alarm or rss thread would continue normally once the
crash handling thread closes the read exception. However, the memory
snapshot machinery can suspend this thread while its in the process of
waiting for or handling a `ZX_EXCP_THREAD_STARTING` sent by either the
rss or alarm thread. If this is suspended first, then we attempt to
suspend either the alarm or rss thread while they're still waiting for
the crash handling thread to handle its exception, we will freeze
waiting for those threads to give the suspend signal, which they won't
because they're blocked on waiting for the exception handler. This is
the deadlock.
Until there's a way for the memory snapshot machinery to suspend the
thread while it's stuck on an exception, then we can work around this in
the meantime by just ensuring the alarm and rss threads start normally
via signals on the initial startup path. I can assert locally the
freezing doesn't occur after 6000 runs where prior we would see it every
couple hundred runs. Note this type of issue can arise again if the
fuzzing test launches any dangling threads that happen to not start yet.
One of the recommendations for writing a fuzz test is that the test may
launch threads, but they should be joined by the end of the test
(https://llvm.org/docs/LibFuzzer.html#fuzz-target), so hopefully we
won't see this type of bug rise frequently from fuzz tests. More
broadly, this can also arise if any process launches its own debugger
via `ZX_EXCEPTION_CHANNEL_DEBUGGER`, but I would think in practice this
isn't very likely to happen.
More context in https://fxbug.dev/436923423.
---------
Co-authored-by: Petr Hosek <phosek@google.com>