This reapplies 6d72bf47606, which was reverted in 57447d3ddf to investigate
build failures, e.g. https://lab.llvm.org/buildbot/#/builders/3/builds/10114.

The original patch contained an invalid unused friend declaration of
std::make_shared. This has been removed.

This reverts commit 6d72bf47606c2a288b911d682fd96129c9c1466d while I fix bot failures.

Also adds a new IdleTask type and updates DynamicThreadPoolTaskDispatcher to
schedule IdleTasks whenever the total number of threads running is less than
the maximum number of MaterializationThreads.

A SimpleLazyReexportsSpeculator instance maintains a list of speculation
suggestions ((JITDylib, Function) pairs) and registered lazy reexports. When
speculation opportunities are available (having been added via
addSpeculationSuggestions or when lazy reexports were created) it schedules
an IdleTask that triggers the next speculative lookup as soon as resources
are available. Speculation suggestions are processed first, followed by
lookups for lazy reexport bodies. A callback can be registered at object
construction time to record lazy reexport executions as they happen, and these
executions can be fed back into the speculator as suggestions on subsequent
executions.

The llvm-jitlink tool is updated to support speculation when lazy linking is
used via three new arguments:

 -speculate=[none|simple] : When the 'simple' value is specified a
                            SimpleLazyReexportsSpeculator instances is used
                            for speculation.

 -speculate-order <path> : Specifies a path to a CSV containing
                           (jit dylib name, function name) triples to use
                           as speculative suggestions in the current run.

 -record-lazy-execs <path> : Specifies a path in which to record lazy function
                             executions as a CSV of (jit dylib name, function
                             name) pairs, suitable for use with
                             -speculate-order.

The same path can be passed to -speculate-order and -record-lazy-execs, in
which case the file will be overwritten at the end of the execution.

No testcase yet: Speculative linking is difficult to test (since by definition
execution behavior should be unaffected by speculation) and this is an new
prototype of the concept*. Tests will be added in the future once the interface
and behavior settle down.

* An earlier implementation of the speculation concept can be found in
  llvm/include/llvm/ExecutionEngine/Orc/Speculation.h. Both systems have the
  same goal (hiding compilation latency) but different mechanisms. This patch
  relies entirely on information available in the controller, where the old
  system could receive additional information from the JIT'd runtime via
  callbacks. I aim to combine the two in the future, but want to gain more
  practical experience with speculation first.

Threads created by DynamicThreadPoolTaskDispatcher::dispatch had been holding a
unique_ptr to the most recent Task, meaning that the Task would be destroyed
when the thread object was destroyed, but this would happen *after* the thread
signaled the Dispatcher that it was finished. This could cause
DynamicThreadPoolTaskDispatcher::shutdown to return (and consequently
ExecutionSession to be destroyed) before all Tasks were destroyed, with Task
destructors accessing ExecutionSession and related objects after they were
freed.

The fix is to reset the Task pointer immediately after it is run to trigger
cleanup, *then* (if there are no other tasks to run) signal the Dispatcher that
the thread is finished.

This patch also updates DynamicThreadPoolTaskDispatcher::dispatch to reject any
new Tasks dispatched after DynamicThreadPoolTaskDispatcher::shutdown is called.

Avoids waking threads when the condition to proceed won't be met. NFC.

Without these explicit includes, removing other headers, who implicitly
include llvm-config.h, may have non-trivial side effects.

This re-applies 6094b3b7db7, which was reverted in e7efd37c229 (and before that
in 1effa19de24) due to bot failures.

The test failures were fixed by having SelfExecutorProcessControl use an
InPlaceTaskDispatcher by default, rather than a DynamicThreadPoolTaskDispatcher.
This shouldn't be necessary (and indicates a concurrency issue elsewhere), but
InPlaceTaskDispatcher is a less surprising default, and better matches the
existing behavior (compilation on current thread by default), so the change
seems reasonable. I've filed https://github.com/llvm/llvm-project/issues/89870
to investigate the concurrency issue as a follow-up.

Coding my way home: 6.25133S 127.94177W

This reverts commit 1effa19de24 while I investigate the test failure at
https://lab.llvm.org/buildbot/#/builders/285/builds/888.

This re-applies 6094b3b7db7, which was reverted in a28557aadd8 due to broken
bots. As far as I can tell all failures were due to a missing #include <deque>,
which has been adedd in this commit.

This reverts commit 6094b3b7db7eab8318b9d30dec2691d231c7bdff.

Multiple bots are broken.