diff options
| author | Florian Hahn <flo@fhahn.com> | 2025-11-12 15:11:00 +0000 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2025-11-12 15:11:00 +0000 |
| commit | 62d1a080e69e3c5e98840e000135afa7c688a77b (patch) | |
| tree | 37804bcf34b0dd57161b2c01ac610b9896db89f4 /llvm/lib/Transforms/Vectorize/LoopVectorize.cpp | |
| parent | 1e4b9e4059e12924b1ad38bd127cdbdb57e3c488 (diff) | |
[LV] Use ExtractLane(LastActiveLane, V) live outs when tail-folding. (#149042)
Building on top of https://github.com/llvm/llvm-project/pull/148817,
introduce a new abstract LastActiveLane opcode that gets lowered to
Not(Mask) → FirstActiveLane(NotMask) → Sub(result, 1).
When folding the tail, update all extracts for uses outside the loop the
extract the value of the last actice lane.
See also https://github.com/llvm/llvm-project/issues/148603
PR: https://github.com/llvm/llvm-project/pull/149042
Diffstat (limited to 'llvm/lib/Transforms/Vectorize/LoopVectorize.cpp')
| -rw-r--r-- | llvm/lib/Transforms/Vectorize/LoopVectorize.cpp | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp index 5cc01a9f974b..b9d4ff41c075 100644 --- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp +++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp @@ -8895,7 +8895,8 @@ void LoopVectorizationPlanner::adjustRecipesForReductions( if (FinalReductionResult == U || Parent->getParent()) continue; U->replaceUsesOfWith(OrigExitingVPV, FinalReductionResult); - if (match(U, m_ExtractLastElement(m_VPValue()))) + if (match(U, m_CombineOr(m_ExtractLastElement(m_VPValue()), + m_ExtractLane(m_VPValue(), m_VPValue())))) cast<VPInstruction>(U)->replaceAllUsesWith(FinalReductionResult); } |