This patch enables the multi-group xnack replay mode by
configuring the hardware MODE register at kernel entry.
This aligns the hardware behavior with the compiler's
existing multi-group s_wait_xcnt insertion logic.

In case there is an dynamic alloca / an alloca which is not in the entry
block, cs.chain functions do not set up an FP, but are reported to need
one. This results in a failed assertion in
`SIFrameLowering::emitPrologue()` (Assertion `(!HasFP || FPSaved) &&
"Needed to save FP but didn't save it anywhere"' failed.) This commit
changes `hasFPImpl` so that the need for an SP in a cs.chain function
does not directly imply the need for an FP anymore.

This LLVM defect was identified via the AMD Fuzzing project.

I was a bit too eager to remove the SI_WHOLE_WAVE_FUNC_SETUP instruction
during prolog emission. Erasing it invalidates MBBI, which in some cases
is still needed outside of `emitCSRSpillStores`.

Do the erasing at the end of prolog insertion instead.

It's nonsensical to call constrainRegClass on a physical register,
and we should not see virtual registers here.

This splits out "ScalablePredicateVector" from the "ScalableVector"
StackID this is primarily to allow easy differentiation between vectors
and predicates (without inspecting instructions).

This new stack ID is not used in many places yet, but will be used in a
later patch to mark stack slots that are known to contain predicates.

Co-authored-by: Kerry McLaughlin <kerry.mclaughlin@arm.com>

Create utility mechanism for finding wave size dependent opcodes used to
manipulate exec/lane masks.

Support tail calls to whole wave functions (trivial) and from whole wave
functions (slightly more involved because we need a new pseudo for the
tail call return, that patches up the EXEC mask).

Move the expansion of whole wave function return pseudos (regular and
tail call returns) to prolog epilog insertion, since that's where we
patch up the EXEC mask.

This is a baseline support, it is not useable yet.

Fixes:
```
*** Bad machine code: Using an undefined physical register ***
- function:    widget
- basic block: %bb.0 bb (0x564092cbe140)
- instruction: $vgpr63 = V_ACCVGPR_READ_B32_e64 killed $agpr13, implicit $exec
- operand 1:   killed $agpr13
LLVM ERROR: Found 1 machine code errors.
```

The detailed sequence of events that led to this assert:

1. MachineVerifier fails because `$agpr13` is not defined on line 19
below:

   ``` 1: bb.0.bb:
    2:   successors: %bb.1(0x80000000); %bb.1(100.00%)
    3:   liveins: $agpr14, $agpr15, $sgpr12, $sgpr13, $sgpr14, \
    4:       $sgpr15, $sgpr30, $sgpr31, $sgpr34, $sgpr35, \
    5:       $sgpr36, $sgpr37, $sgpr38, $sgpr39, $sgpr48, \
    6:       $sgpr49, $sgpr50, $sgpr51, $sgpr52, $sgpr53, \
    7:       $sgpr54, $sgpr55, $sgpr64, $sgpr65, $sgpr66, \
    8:       $sgpr67, $sgpr68, $sgpr69, $sgpr70, $sgpr71, \
    9:       $sgpr80, $sgpr81, $sgpr82, $sgpr83, $sgpr84, \
   10:       $sgpr85, $sgpr86, $sgpr87, $sgpr96, $sgpr97, \
   11:       $sgpr98, $sgpr99, $vgpr0, $vgpr31, $vgpr40, $vgpr41, \
   12:       $sgpr4_sgpr5, $sgpr6_sgpr7, $sgpr8_sgpr9, \
   13:       $sgpr10_sgpr11
   14:   $sgpr16 = COPY $sgpr33
   15:   $sgpr33 = frame-setup COPY $sgpr32
   16:   $sgpr18_sgpr19 = S_XOR_SAVEEXEC_B64 -1, \
   17:       implicit-def $exec, implicit-def dead $scc, \
   18:       implicit $exec
   19:   $vgpr63 = V_ACCVGPR_READ_B32_e64 killed $agpr13, \
   20:       implicit $exec
   21:   BUFFER_STORE_DWORD_OFFSET killed $vgpr63, \
   22:       $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr33, 0, 0, 0, \
   23:       implicit $exec :: (store (s32) into %stack.38, \
   24:       addrspace 5)
   25:   ...
   26:   $vgpr43 = IMPLICIT_DEF
   27:   $vgpr43 = SI_SPILL_S32_TO_VGPR $sgpr15, 0, \
   28:       killed $vgpr43(tied-def 0)
   29:   $vgpr43 = SI_SPILL_S32_TO_VGPR $sgpr14, 1, \
   30:       killed $vgpr43(tied-def 0)
   31:   $sgpr100_sgpr101 = S_OR_SAVEEXEC_B64 -1, \
   32:       implicit-def $exec, implicit-def dead $scc, \
   33:       implicit $exec
   34:   renamable $agpr13 = COPY killed $vgpr43, implicit $exec
   ```

2. That instruction is created by
[`emitCSRSpillStores`](https://github.com/llvm/llvm-project/blob/d599bdeaa49d7a2b1246328630328d23ddda5a47/llvm/lib/Target/AMDGPU/SIFrameLowering.cpp#L977)
(called by
[`SIFrameLowering::emitPrologue`](https://github.com/llvm/llvm-project/blob/d599bdeaa49d7a2b1246328630328d23ddda5a47/llvm/lib/Target/AMDGPU/SIFrameLowering.cpp#L1122))
because `$agpr13` is in `WWMSpills`.

   See lines 982, 998, and 993 below.

   ```
977: // Spill Whole-Wave Mode VGPRs. Save only the inactive lanes of the
scratch
978: // registers. However, save all lanes of callee-saved VGPRs. Due to
this, we
   979:   // might end up flipping the EXEC bits twice.
   980:   Register ScratchExecCopy;
981: SmallVector<std::pair<Register, int>, 2> WWMCalleeSavedRegs,
WWMScratchRegs;
982: FuncInfo->splitWWMSpillRegisters(MF, WWMCalleeSavedRegs,
WWMScratchRegs);
   983:   if (!WWMScratchRegs.empty())
   984:     ScratchExecCopy =
   985:         buildScratchExecCopy(LiveUnits, MF, MBB, MBBI, DL,
986: /*IsProlog*/ true, /*EnableInactiveLanes*/ true);
   987:
   988:   auto StoreWWMRegisters =
   989:       [&](SmallVectorImpl<std::pair<Register, int>> &WWMRegs) {
   990:         for (const auto &Reg : WWMRegs) {
   991:           Register VGPR = Reg.first;
   992:           int FI = Reg.second;
993: buildPrologSpill(ST, TRI, *FuncInfo, LiveUnits, MF, MBB, MBBI, DL,
   994:                            VGPR, FI, FrameReg);
   995:         }
   996:       };
   997:
   998:   StoreWWMRegisters(WWMScratchRegs);
   ```

3. `$agpr13` got added to `WWMSpills` by
[`SILowerWWMCopies::run`](https://github.com/llvm/llvm-project/blob/59a7185dd9d69cbf737a98f5c2d1cf3d456bee03/llvm/lib/Target/AMDGPU/SILowerWWMCopies.cpp#L137)
as it processed the `WWM_COPY` on line 3 below (corresponds to line 34
above in point #_1_):

   ```
1: %45:vgpr_32 = SI_SPILL_S32_TO_VGPR $sgpr15, 0, %45:vgpr_32(tied-def
0)
2: %45:vgpr_32 = SI_SPILL_S32_TO_VGPR $sgpr14, 1, %45:vgpr_32(tied-def
0)
   3: %44:av_32 = WWM_COPY %45:vgpr_32
   ```

Whole wave functions are functions that will run with a full EXEC mask.
They will not be invoked directly, but instead will be launched by way
of a new intrinsic, `llvm.amdgcn.call.whole.wave` (to be added in
a future patch). These functions are meant as an alternative to the
`llvm.amdgcn.init.whole.wave` or `llvm.amdgcn.strict.wwm` intrinsics.

Whole wave functions will set EXEC to -1 in the prologue and restore the
original value of EXEC in the epilogue. They must have a special first
argument, `i1 %active`, that is going to be mapped to EXEC. They may
have either the default calling convention or amdgpu_gfx. The inactive
lanes need to be preserved for all registers used, active lanes only for
the CSRs.

At the IR level, arguments to a whole wave function (other than
`%active`) contain poison in their inactive lanes. Likewise, the return
value for the inactive lanes is poison.

This patch contains the following work:
* 2 new pseudos, SI_SETUP_WHOLE_WAVE_FUNC and SI_WHOLE_WAVE_FUNC_RETURN
  used for managing the EXEC mask. SI_SETUP_WHOLE_WAVE_FUNC will return
  a SReg_1 representing `%active`, which needs to be passed into
  SI_WHOLE_WAVE_FUNC_RETURN.
* SelectionDAG support for generating these 2 new pseudos and the
  special handling of %active. Since the return may be in a different
  basic block, it's difficult to add the virtual reg for %active to
  SI_WHOLE_WAVE_FUNC_RETURN, so we initially generate an IMPLICIT_DEF
  which is later replaced via a custom inserter.
* Expansion of the 2 pseudos during prolog/epilog insertion. PEI also
  marks any used VGPRs as WWM registers, which are then spilled and
  restored with the usual logic.

Future patches will include the `llvm.amdgcn.call.whole.wave` intrinsic
and a lot of optimization work (especially in order to reduce spills
around function calls).

---------

Co-authored-by: Matt Arsenault <Matthew.Arsenault@amd.com>
Co-authored-by: Shilei Tian <i@tianshilei.me>