summaryrefslogtreecommitdiff
path: root/llvm/lib/Target/AMDGPU/SIInsertSkips.cpp
AgeCommit message (Collapse)Author
2021-03-20[AMDGPU] Rename SIInsertSkips PassCarl Ritson
Pass no longer handles skips. Pass now removes unnecessary unconditional branches and lowers early termination branches. Hence rename to SILateBranchLowering. Move code to handle returns to epilog from SIPreEmitPeephole into SILateBranchLowering. This means SIPreEmitPeephole only contains optional optimisations, and all required transforms are in SILateBranchLowering. Reviewed By: arsenm Differential Revision: https://reviews.llvm.org/D98915
2021-03-09[AMDGPU] Remove unused function opcodeEmitsNoInsts()Ruiling Song
This was missed in the patch D97545, and cause buildbot failure. Reviewed by: critson Differential Revision: https://reviews.llvm.org/D98229
2021-03-09[AMDGPU] Remove SI_MASK_BRANCHRuiling Song
This is already deprecated, so remove code working on this. Also update the tests by using S_CBRANCH_EXECZ instead of SI_MASK_BRANCH. Reviewed By: foad Differential Revision: https://reviews.llvm.org/D97545
2021-02-11[AMDGPU] Move kill lowering to WQM pass and add live mask trackingCarl Ritson
Move implementation of kill intrinsics to WQM pass. Add live lane tracking by updating a stored exec mask when lanes are killed. Use live lane tracking to enable early termination of shader at any point in control flow. Reviewed By: piotr Differential Revision: https://reviews.llvm.org/D94746
2021-01-20[NFC][AMDGPU] Split AMDGPUSubtarget.h to R600 and GCN subtargetsdfukalov
... to reduce headers dependency. Reviewed By: rampitec, arsenm Differential Revision: https://reviews.llvm.org/D95036
2021-01-13[AMDGPU] Add SI_EARLY_TERMINATE_SCC0 for early terminating shaderCarl Ritson
Add pseudo instruction to allow early termination of pixel shader anywhere based on the value of SCC. The intention is to use this when a mask of live lanes is updated, e.g. live lanes in WQM pass. This facilitates early termination of shaders even when EXEC is incomplete, e.g. in non-uniform control flow. Reviewed By: foad Differential Revision: https://reviews.llvm.org/D88777
2021-01-07[NFC][AMDGPU] Reduce include files dependency.dfukalov
Reviewed By: rampitec Differential Revision: https://reviews.llvm.org/D93813
2020-12-04[AMDGPU] Use llvm::is_contained (NFC)Kazu Hirata
2020-11-12[AMDGPU] Define and use names for export targets. NFC.Jay Foad
Differential Revision: https://reviews.llvm.org/D91289
2020-10-06[AMDGPU] SIInsertSkips: Refactor early exit block creationCarl Ritson
Refactor exit block creation to a single call ensureEarlyExitBlock. Add support for generating an early exit block which clears the exec mask, but only add this instruction when required. These changes are to facilitate adding more forms of early termination for PS shaders in the near future. Reviewed By: nhaehnle Differential Revision: https://reviews.llvm.org/D88775
2020-10-02Handle unused variable without assertsTres Popp
2020-10-02[AMDGPU] SIInsertSkips: Tidy block splitting to use splitAtCarl Ritson
Convert to use new MachineBasicBlock splitAt function. Place code in splitBlock function for reuse in future changes. Should yield no functional change. Reviewed By: arsenm Differential Revision: https://reviews.llvm.org/D88537
2020-07-03[AMDGPU] Insert PS early exit at end of control flowCarl Ritson
Exit early if the exec mask is zero at the end of control flow. Mark the ends of control flow during control flow lowering and convert these to exits during the insert skips pass. Reviewed By: nhaehnle Differential Revision: https://reviews.llvm.org/D82737
2020-07-03Revert "[AMDGPU] Insert PS early exit at end of control flow"Carl Ritson
This reverts commit 2bfcacf0ad362956277a1c2c9ba00ddc453a42ce. There appears to be an issue to analysis preservation.
2020-07-03[AMDGPU] Insert PS early exit at end of control flowCarl Ritson
Exit early if the exec mask is zero at the end of control flow. Mark the ends of control flow during control flow lowering and convert these to exits during the insert skips pass. Reviewed By: nhaehnle Differential Revision: https://reviews.llvm.org/D82737
2020-07-03[AMDGPU] Unify early PS termination blocksCarl Ritson
Generate a single early exit block out-of-line and branch to this if all lanes are killed. This avoids branching if lanes are active. Reviewed By: nhaehnle Differential Revision: https://reviews.llvm.org/D82641
2020-06-29[AMDGPU] Moving SI_RETURN_TO_EPILOG handling out of SIInsertSkips.Christudasan Devadasan
For now, moving it to SIPreEmitPeephole. Should find a right place to have this code. Reviewed By: nhaehnle Differential revision: https://reviews.llvm.org/D77544
2020-03-25[AMDGPU] Add SIPreEmitPeephole pass.cdevadas
This pass can handle all the optimization opportunities found just before code emission. Presently it includes the handling of vcc branch optimization that was handled earlier in SIInsertSkips. Reviewed By: arsenm Differential Revision: https://reviews.llvm.org/D76712
2020-02-26Full fix for "AMDGPU/SIInsertSkips: Fix the determination of whether ↵Nicolai Hähnle
early-exit-after-kill is possible" (hopefully) Properly preserve the MachineDominatorTree in all cases. Change-Id: I54cf0c0a20934168a356920ba8ed5097a93c4131
2020-02-26Quick fix for bot failure on "AMDGPU/SIInsertSkips: Fix the determination of ↵Nicolai Hähnle
whether early-exit-after-kill is possible" Apparently the dominator tree update is incorrect, will investigate. Change-Id: Ie76f8d11b22a552af1f098c893773f3d85e02d4f
2020-02-26AMDGPU/SIInsertSkips: Fix the determination of whether early-exit-after-kill ↵Nicolai Hähnle
is possible Summary: The old code made some incorrect assumptions about the order in which basic blocks are laid out in a function. This could lead to incorrect early-exits, especially when kills occurred inside of loops. The new approach is to check whether the point where the conditional kill occurs dominates all reachable code. If that is the case, there cannot be any other threads in the wave that are waiting to rejoin at a later point in the CFG, i.e. if exec=0 at that point, then all threads really are dead and we can exit the wave. Make some other minor cleanups to the pass while we're at it. v2: preserve the dominator tree Reviewers: arsenm, cdevadas, foad, critson Subscribers: kzhuravl, jvesely, wdng, yaxunl, dstuttard, tpr, t-tye, hiraditya, kerbowa, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D74908 Change-Id: Ia0d2b113ac944ad642d1c622b6da1b20aa1aabcc
2020-01-22Resubmit: [AMDGPU] Invert the handling of skip insertion.cdevadas
The current implementation of skip insertion (SIInsertSkip) makes it a mandatory pass required for correctness. Initially, the idea was to have an optional pass. This patch inserts the s_cbranch_execz upfront during SILowerControlFlow to skip over the sections of code when no lanes are active. Later, SIRemoveShortExecBranches removes the skips for short branches, unless there is a sideeffect and the skip branch is really necessary. This new pass will replace the handling of skip insertion in the existing SIInsertSkip Pass. Differential revision: https://reviews.llvm.org/D68092
2020-01-21Revert "[AMDGPU] Invert the handling of skip insertion."Nicolai Hähnle
This reverts commit 0dc6c249bffac9f23a605ce4e42a84341da3ddbd. The commit is reported to cause a regression in piglit/bin/glsl-vs-loop for Mesa.
2020-01-15[AMDGPU] Invert the handling of skip insertion.cdevadas
The current implementation of skip insertion (SIInsertSkip) makes it a mandatory pass required for correctness. Initially, the idea was to have an optional pass. This patch inserts the s_cbranch_execz upfront during SILowerControlFlow to skip over the sections of code when no lanes are active. Later, SIRemoveShortExecBranches removes the skips for short branches, unless there is a sideeffect and the skip branch is really necessary. This new pass will replace the handling of skip insertion in the existing SIInsertSkip Pass. Differential revision: https://reviews.llvm.org/D68092
2019-06-16[AMDGPU] gfx10 conditional registers handlingStanislav Mekhanoshin
This is cpp source part of wave32 support, excluding overriden getRegClass(). Differential Revision: https://reviews.llvm.org/D63351 llvm-svn: 363513
2019-06-07AMDGPU: Don't count mask branch pseudo towards skip thresholdMatt Arsenault
llvm-svn: 362761
2019-06-07AMDGPU: Insert skips for blocks with FLATMatt Arsenault
This already forced a skip for VMEM, so it should also be done for flat. I'm somewhat skeptical about the benefit of this though. llvm-svn: 362760
2019-06-06AMDGPU: Insert skip branches over return blocksMatt Arsenault
SIInsertSkips really doesn't understand the control flow, and makes very stupid assumptions about the block layout. This was able to get away with not skipping return blocks, since usually after structurization there is only one placed at the end of the function. Tail duplication can break this assumption. llvm-svn: 362754
2019-06-03AMDGPU/GFX10: V_CMPX_xxx instructions still have an omod operandNicolai Haehnle
Summary: Change-Id: If6ee98e4a723b643bc37254fc6ef8b3812db16da Reviewers: rampitec Subscribers: arsenm, kzhuravl, jvesely, wdng, yaxunl, dstuttard, tpr, t-tye, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D62720 Change-Id: Id547ef152b2f92b24dc1c0efbf7e4467c4fb4b6e llvm-svn: 362390
2019-04-26[AMDGPU] gfx1010 VOPC implementationStanislav Mekhanoshin
Differential Revision: https://reviews.llvm.org/D61208 llvm-svn: 359358
2019-04-17AMDGPU: Force skip over SMRD, VMEM and s_waitcnt instructionsRhys Perry
Summary: This fixes a large Dawn of War 3 performance regression with RADV from Mesa 19.0 to master which was caused by creating less code in some branches. Reviewers: arsen, nhaehnle Reviewed By: nhaehnle Subscribers: arsenm, kzhuravl, jvesely, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D60824 llvm-svn: 358592
2019-03-12[AMDGPU] Add support for immediate operand for S_ENDPGMDavid Stuttard
Summary: Add support for immediate operand in S_ENDPGM Change-Id: I0c56a076a10980f719fb2a8f16407e9c301013f6 Reviewers: alexshap Subscribers: qcolombet, arsenm, kzhuravl, jvesely, wdng, nhaehnle, yaxunl, tpr, t-tye, eraman, arphaman, Petar.Avramovic, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D59213 llvm-svn: 355902
2019-01-19Update the file headers across all of the LLVM projects in the monorepoChandler Carruth
to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
2018-11-16AMDGPU: Fix analyzeBranch failing with pseudoterminatorsMatt Arsenault
If a block had one of the _term instructions used for gluing exec modifying instructions to the end of the block, analyzeBranch would fail, preventing the verifier from catching a broken successor list. llvm-svn: 347027
2018-11-12[AMDGPU] Optimize S_CBRANCH_VCC[N]Z -> S_CBRANCH_EXEC[N]ZStanislav Mekhanoshin
Sometimes after basic block placement we end up with a code like: sreg = s_mov_b64 -1 vcc = s_and_b64 exec, sreg s_cbranch_vccz This happens as a join of a block assigning -1 to a saved mask and another block which consumes that saved mask with s_and_b64 and a branch. This is essentially a single s_cbranch_execz instruction when moved into a single new basic block. Differential Revision: https://reviews.llvm.org/D54164 llvm-svn: 346690
2018-07-30AMDGPU: Force skip over s_sendmsg and exp instructionsNicolai Haehnle
Summary: These instructions interact with hardware blocks outside the shader core, and they can have "scalar" side effects even when EXEC = 0. We don't want these scalar side effects to occur when all lanes want to skip these instructions, so always add the execz skip branch instruction for basic blocks that contain them. Also ensure that we skip scalar stores / atomics, though we don't code-gen those yet. Reviewers: arsenm, rampitec Subscribers: kzhuravl, wdng, yaxunl, dstuttard, tpr, t-tye, llvm-commits Differential Revision: https://reviews.llvm.org/D48431 Change-Id: Ieaeb58352e2789ffd64745603c14970c60819d44 llvm-svn: 338235
2018-07-11AMDGPU: Refactor Subtarget classesTom Stellard
Summary: This is a follow-up to r335942. - Merge SISubtarget into AMDGPUSubtarget and rename to GCNSubtarget - Rename AMDGPUCommonSubtarget to AMDGPUSubtarget - Merge R600Subtarget::Generation and GCNSubtarget::Generation into AMDGPUSubtarget::Generation. Reviewers: arsenm, jvesely Subscribers: kzhuravl, wdng, nhaehnle, yaxunl, dstuttard, tpr, t-tye, javed.absar, llvm-commits Differential Revision: https://reviews.llvm.org/D49037 llvm-svn: 336851
2018-05-22AMDGPU: Remove #include "MCTargetDesc/AMDGPUMCTargetDesc.h" from common headersTom Stellard
Summary: MCTargetDesc/AMDGPUMCTargetDesc.h contains enums for all the instuction and register defintions, which are huge so we only want to include them where needed. This will also make it easier if we want to split the R600 and GCN definitions into separate tablegenerated files. I was unable to remove AMDGPUMCTargetDesc.h from SIMachineFunctionInfo.h because it uses some enums from the header to initialize default values for the SIMachineFunction class, so I ended up having to remove includes of SIMachineFunctionInfo.h from headers too. Reviewers: arsenm, nhaehnle Reviewed By: nhaehnle Subscribers: MatzeB, kzhuravl, wdng, yaxunl, dstuttard, tpr, t-tye, javed.absar, llvm-commits Differential Revision: https://reviews.llvm.org/D46272 llvm-svn: 332930
2018-05-01Remove \brief commands from doxygen comments.Adrian Prantl
We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
2018-02-23Fix compiler warning introduced in r325931. NFC.Geoff Berry
llvm-svn: 325938
2018-02-23[MachineOperand][Target] MachineOperand::isRenamable semantics changesGeoff Berry
Summary: Add a target option AllowRegisterRenaming that is used to opt in to post-register-allocation renaming of registers. This is set to 0 by default, which causes the hasExtraSrcRegAllocReq/hasExtraDstRegAllocReq fields of all opcodes to be set to 1, causing MachineOperand::isRenamable to always return false. Set the AllowRegisterRenaming flag to 1 for all in-tree targets that have lit tests that were effected by enabling COPY forwarding in MachineCopyPropagation (AArch64, AMDGPU, ARM, Hexagon, Mips, PowerPC, RISCV, Sparc, SystemZ and X86). Add some more comments describing the semantics of the MachineOperand::isRenamable function and how it is set and maintained. Change isRenamable to check the operand's opcode hasExtraSrcRegAllocReq/hasExtraDstRegAllocReq bit directly instead of relying on it being consistently reflected in the IsRenamable bit setting. Clear the IsRenamable bit when changing an operand's register value. Remove target code that was clearing the IsRenamable bit when changing registers/opcodes now that this is done conservatively by default. Change setting of hasExtraSrcRegAllocReq in AMDGPU target to be done in one place covering all opcodes that have constant pipe read limit restrictions. Reviewers: qcolombet, MatzeB Subscribers: aemerson, arsenm, jyknight, mcrosier, sdardis, nhaehnle, javed.absar, tpr, arichardson, kristof.beyls, kbarton, fedor.sergeev, asb, rbar, johnrusso, simoncook, jordy.potman.lists, apazos, sabuasal, niosHD, escha, nemanjai, llvm-commits Differential Revision: https://reviews.llvm.org/D43042 llvm-svn: 325931
2018-01-30[AMDGPU] isRenamable fixes to support copy forwardingGeoff Berry
Mark more opcodes as hasExtraSrcRegAllocReq so that their operands will be marked as not renamable, to avoid copy forwarding violating the constraint that only one operand may use the constant bus. These changes fix a few mis-compiles when copy forwarding is enabled in MachineCopyPropagation by D41835 (and were reviewed as part of that change). llvm-svn: 323794
2018-01-29AMDGPU: Allow a SGPR for the conditional KILL operandMarek Olsak
Patch by: Bas Nieuwenhuizen Just use the _e64 variant if needed. This should be possible as per def : Pat < (int_amdgcn_kill (i1 (setcc f32:$src, InlineFPImm<f32>:$imm, cond:$cond))), (SI_KILL_F32_COND_IMM_PSEUDO $src, (bitcast_fpimm_to_i32 $imm), (cond_as_i32imm $cond)) > ; I don't think we can get an immediate for the other operand for which we need the second 32-bit word. https://reviews.llvm.org/D42302 llvm-svn: 323706
2017-12-15MachineFunction: Return reference from getFunction(); NFCMatthias Braun
The Function can never be nullptr so we can return a reference. llvm-svn: 320884
2017-10-24AMDGPU: Add new intrinsic llvm.amdgcn.kill(i1)Marek Olsak
Summary: Kill the thread if operand 0 == false. llvm.amdgcn.wqm.vote can be applied to the operand. Also allow kill in all shader stages. Reviewers: arsenm, nhaehnle Subscribers: kzhuravl, wdng, yaxunl, dstuttard, tpr, t-tye, llvm-commits Differential Revision: https://reviews.llvm.org/D38544 llvm-svn: 316427
2017-10-03[AMDGPU] Avoid predicated execution of the basic blocks containing scalarAlexander Timofeev
instructions. Differential revision: https://reviews.llvm.org/D38293 llvm-svn: 314828
2017-03-21AMDGPU: Rename SI_RETURNMatt Arsenault
This is used for a specific type of return to a shader part's epilog code. Rename to try avoiding confusion from a true call's return. llvm-svn: 298452
2017-01-24AMDGPU: Remove spurious out branches after a killMatt Arsenault
The sequence like this: v_cmpx_le_f32_e32 vcc, 0, v0 s_branch BB0_30 s_cbranch_execnz BB0_30 ; BB#29: exp null off, off, off, off done vm s_endpgm BB0_30: ; %endif110 is likely wrong. The s_branch instruction will unconditionally jump to BB0_30 and the skip block (exp done + endpgm) inserted for performing the kill instruction will never be executed. This results in a GPU hang with Star Ruler 2. The s_branch instruction is added during the "Control Flow Optimizer" pass which seems to re-organize the basic blocks, and we assume that SI_KILL_TERMINATOR is always the last instruction inside a basic block. Thus, after inserting a skip block we just go to the next BB without looking at the subsequent instructions after the kill, and the s_branch op is never removed. Instead, we should remove the unconditional out branches and let skip the two instructions if the exec mask is non-zero. This patch fixes the GPU hang and doesn't introduce any regressions with "make check". Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=99019 Patch by Samuel Pitoiset <samuel.pitoiset@gmail.com> llvm-svn: 292985
2017-01-21[AMDGPU] Fix some Clang-tidy modernize and Include What You Use warnings; ↵Eugene Zelenko
other minor fixes (NFC). llvm-svn: 292688
2017-01-13[CodeGen] Rename MachineInstrBuilder::addOperand. NFCDiana Picus
Rename from addOperand to just add, to match the other method that has been added to MachineInstrBuilder for adding more than just 1 operand. See https://reviews.llvm.org/D28057 for the whole discussion. Differential Revision: https://reviews.llvm.org/D28556 llvm-svn: 291891
generated by cgit v1.2.3 (git 2.25.1) at 2026-10-22 19:31:14 +0000