summaryrefslogtreecommitdiff
path: root/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
AgeCommit message (Collapse)Author
2025-06-20[AMDGPU] Don't insert wait instructions that are not supported by gfx1250 ↵Stanislav Mekhanoshin
(#145084) No tests yet, but it will allow further tests not to be polluted with these waits.
2025-05-28Warn on misuse of DiagnosticInfo classes that hold Twines (#137397)Justin Bogner
This annotates the `Twine` passed to the constructors of the various DiagnosticInfo subclasses with `[[clang::lifetimebound]]`, which causes us to warn when we would try to print the twine after it had already been destructed. We also update `DiagnosticInfoUnsupported` to hold a `const Twine &` like all of the other DiagnosticInfo classes, since this warning allows us to clean up all of the places where it was being used incorrectly.
2025-03-12[AMDGPU][NPM] Port SIMemoryLegalizer to NPM (#130060)Akshat Oke
2025-02-19[AMDGPU] Remove FeatureForceStoreSC0SC1 (#126878)Fabian Ritter
This was only used for gfx940 and gfx941, which have since been removed. For SWDEV-512631
2025-02-19[AMDGPU] Replace gfx940 and gfx941 with gfx942 in llvm (#126763)Fabian Ritter
gfx940 and gfx941 are no longer supported. This is one of a series of PRs to remove them from the code base. This PR removes all non-documentation occurrences of gfx940/gfx941 from the llvm directory, and the remaining occurrences in clang. Documentation changes will follow. For SWDEV-512631
2024-10-07[AMDGPU] Only emit SCOPE_SYS global_wb (#110636)Pierre van Houtryve
global_wb with scopes lower than SCOPE_SYS is unnecessary for correctness. I was initially optimistic they would be very cheap no-ops but they can actually be quite expensive so let's avoid them.
2024-09-09[AMDGPU] Document & Finalize GFX12 Memory Model (#98599)Pierre van Houtryve
Documents the memory model implemented as of #98591, with some fixes/optimizations to the implementation.
2024-07-22AMDGPU: Query MachineModuleInfo from PM instead of MachineFunction (#99679)Matt Arsenault
2024-07-16[AMDGPU] Fix and add namespace closing comments. NFC.Jay Foad
2024-07-16[AMDGPU] Implement GFX12 Memory Model (#98591)Pierre van Houtryve
- Emit GLOBAL_WB instructions - Reflect synscope on instructions's `scope:` operand Fixes SWDEV-468508 Fixes SWDEV-470735 Fixes SWDEV-468392 Fixes SWDEV-469622
2024-05-27[AMDGPU] Add amdgpu-as MMRA for fences (#78572)Pierre van Houtryve
Using MMRAs, allow `builtin_amdgcn_fence` to emit fences that only target one or more address spaces, instead of fencing all address spaces at once. This is done through a `amdgpu-as` MMRA. Currently focused on OpenCL fences, but can very easily support more AS names and codegen on more than just fences.
2024-03-06[AMDGPU] Handle amdgpu.last.use metadata (#83816)Mirko Brkušanin
Convert !amdgpu.last.use metadata into MachineMemOperand for last use and handle it in SIMemoryLegalizer similar to nontemporal and volatile.
2024-03-04[AMDGPU] Fix setting nontemporal in memory legalizer (#83815)Mirko Brkušanin
Iterator MI can advance in insertWait() but we need original instruction to set temporal hint. Just move it before handling volatile.
2024-02-28AMDGPU/GFX12: Insert waitcnts before stores with scope_sys (#82996)Petar Avramovic
Insert waitcnts for loads and atomics before stores with system scope. Scope is field in instruction encoding and corresponds to desired coherence level in cache hierarchy. Intrinsic stores can set scope in cache policy operand. If volatile keyword is used on generic stores memory legalizer will set scope to system. Generic stores, by default, get lowest scope level. Waitcnts are not required if it is guaranteed that memory is cached. For example vulkan shaders can guarantee this. TODO: implement flag for frontends to give us a hint not to insert waits. Expecting vulkan flag to be implemented as vulkan:private MMRA.
2024-02-13[AMDGPU][SIMemoryLegalizer] Fix order of GL0/1_INV on GFX10/11 (#81450)Pierre van Houtryve
Fixes SWDEV-443292
2024-01-18[AMDGPU] CodeGen for GFX12 S_WAIT_* instructions (#77438)Jay Foad
Update SIMemoryLegalizer and SIInsertWaitcnts to use separate wait instructions per counter (e.g. S_WAIT_LOADCNT) and split VMCNT into separate LOADCNT, SAMPLECNT and BVHCNT counters.
2024-01-08[AMDGPU] Add new cache flushing instructions for GFX12 (#76944)Mirko Brkušanin
Co-authored-by: Diana Picus <Diana-Magda.Picus@amd.com>
2023-12-15[AMDGPU][SIInsertWaitcnts] Do not add s_waitcnt when the counters are known ↵Pierre van Houtryve
to be 0 already (#72830) Co-authored-by: Juan Manuel MARTINEZ CAAMAÑO <juamarti@amd.com>
2023-05-12AMDGPU: Force sc0 and sc1 on stores for gfx940 and gfx941Konstantin Zhuravlyov
Differential Revision: https://reviews.llvm.org/D149986
2023-03-08[AMDGPU] Skip buffer_wbl2 before atomic fence acquireStanislav Mekhanoshin
Memory models for gfx90a and gfx940 do not require buffer_wbl2 before the fence for acquire ordering, but we do insert the full release. Fixes: SWDEV-386785 Differential Revision: https://reviews.llvm.org/D145524
2023-02-07[NFC][TargetParser] Remove llvm/Support/TargetParser.hArchibald Elliott
2022-12-17std::optional::value => operator*/operator->Fangrui Song
value() has undesired exception checking semantics and calls __throw_bad_optional_access in libc++. Moreover, the API is unavailable without _LIBCPP_NO_EXCEPTIONS on older Mach-O platforms (see _LIBCPP_AVAILABILITY_BAD_OPTIONAL_ACCESS). This fixes clang.
2022-12-14[AMDGPU] Stop using make_pair and make_tuple. NFC.Jay Foad
C++17 allows us to call constructors pair and tuple instead of helper functions make_pair and make_tuple. Differential Revision: https://reviews.llvm.org/D139828
2022-12-13[CodeGen] llvm::Optional => std::optionalFangrui Song
2022-12-08[llvm] Use std::nullopt instead of None in comments (NFC)Kazu Hirata
This is part of an effort to migrate from llvm::Optional to std::optional: https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
2022-12-02[Target] Use std::nullopt instead of None (NFC)Kazu Hirata
This patch mechanically replaces None with std::nullopt where the compiler would warn if None were deprecated. The intent is to reduce the amount of manual work required in migrating from Optional to std::optional. This is part of an effort to migrate from llvm::Optional to std::optional: https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
2022-09-02[AMDGPU][NFC] Fix typo in commment: replace SiMemOpInfo by SIMemOpInfoJuan Manuel MARTINEZ CAAMAÑO
2022-07-13[llvm] Use value instead of getValue (NFC)Kazu Hirata
2022-06-25Revert "Don't use Optional::hasValue (NFC)"Kazu Hirata
This reverts commit aa8feeefd3ac6c78ee8f67bf033976fc7d68bc6d.
2022-06-25Don't use Optional::hasValue (NFC)Kazu Hirata
2022-06-20[llvm] Don't use Optional::getValue (NFC)Kazu Hirata
2022-06-10[AMDGPU] Update dlc usage for GFX11Jay Foad
In GFX10 dlc controlled L1 cache bypass. In GFX11 it has been repurposed to control MALL NOALLOC, and glc controls L1 as well as L0 cache bypass. Update the documentation and SIMemoryLegalizer accordingly. Set dlc for nontemporal and volatile accesses. Differential Revision: https://reviews.llvm.org/D127405
2022-03-16Cleanup codegen includesserge-sans-paille
This is a (fixed) recommit of https://reviews.llvm.org/D121169 after: 1061034926 before: 1063332844 Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup Differential Revision: https://reviews.llvm.org/D121681
2022-03-14[AMDGPU] gfx940 memory modelStanislav Mekhanoshin
Differential Revision: https://reviews.llvm.org/D121242
2022-03-11[AMDGPU] gfx940 MUBUF format changesStanislav Mekhanoshin
Differential Revision: https://reviews.llvm.org/D121234
2022-03-10Revert "Cleanup codegen includes"Nico Weber
This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20. Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang, and many LLVM tests, see comments on https://reviews.llvm.org/D121169
2022-03-10Cleanup codegen includesserge-sans-paille
after: 1061034926 before: 1063332844 Differential Revision: https://reviews.llvm.org/D121169
2022-02-18[AMDGPU][NFC] Fix typosSebastian Neubauer
Fix some typos in the amdgpu backend. Differential Revision: https://reviews.llvm.org/D119235
2021-11-26[AMDGPU] Add SIMemoryLegalizer comments to clarify bit usageCarl Ritson
Attempt to further document the intended cache policies requested by different combinations of GLC, SLC and DLC bits. GFX10 non-temporal stores are updated to set GLC. Reviewed By: t-tye Differential Revision: https://reviews.llvm.org/D114351
2021-11-05[AMDGPU] NFC formatting fixes in SIMemoryLegalizerJay Foad
2021-08-10[AMDGPU] Support non-strictly stronger memory orderings in SIMemoryLegalizerTony Tye
C++20 no longer requires the failure memory ordering to be no stronger than the success memory ordering. Adjust assert in AMD GPU SIMemoryLegalizer, and merge instruction memory orderings Add common operation to merge memory orders that allows non strict memory orderings to be combined. Use it in SIMemoryLegalizer and MachineMemOperand::getMergedOrdering. Reviewed By: efriedma, rampitec Differential Revision: https://reviews.llvm.org/D106729
2021-06-30[AMDGPU] Update gfx90a memory model supportTony Tye
Update AMDGPU gfx90a memory model to make coarse grain memory allocations consistent when fine grained system scope atomic acquire and release is performed. Reviewed By: rampitec Differential Revision: https://reviews.llvm.org/D105137
2021-06-21Rename MachineMemOperand::getOrdering -> getSuccessOrdering.Eli Friedman
Since this method can apply to cmpxchg operations, make sure it's clear what value we're actually retrieving. This will help ensure we don't accidentally ignore the failure ordering of cmpxchg in the future. We could potentially introduce a getOrdering() method on AtomicSDNode that asserts the operation isn't cmpxchg, but not sure that's worthwhile. Differential Revision: https://reviews.llvm.org/D103338
2021-04-07[AMDGPU] Update gfx90a memory model supportTony Tye
Reviewed By: rampitec Differential Revision: https://reviews.llvm.org/D100070
2021-03-23[AMDGPU] Only unbundle memory accesses in SIMemoryLegalizerCarl Ritson
This restores previous behaviour and is a step toward removing unbundling entirely. Reviewed By: foad, rampitec Differential Revision: https://reviews.llvm.org/D99061
2021-03-15[AMDGPU] Use single cache policy operandStanislav Mekhanoshin
Replace individual operands GLC, SLC, and DLC with a single cache_policy bitmask operand. This will reduce the number of operands in MIR and I hope the amount of code. These operands are mostly 0 anyway. Additional advantage that parser will accept these flags in any order unlike now. Differential Revision: https://reviews.llvm.org/D96469
2021-02-17[AMDGPU] gfx90a supportStanislav Mekhanoshin
Differential Revision: https://reviews.llvm.org/D96906
2021-02-17[AMDGPU] Correct rmw atomics s_waitcnt generationTony Tye
The AMD GPU SIMemoryLegalizer was using the ordering address space rather than the instruction address space when determining the s_waitcnt to generate to ensure that a read-modify-write atomic has completed. This resulted in additional unnecessary counters being waited on. Differential Revision: https://reviews.llvm.org/D96743
2021-02-15[AMDGPU] Add two TSFlags: IsAtomicNoRtn and IsAtomicRtnStanislav Mekhanoshin
We are using AtomicNoRet map in multiple places to determine if an instruction atomic, rtn or nortn atomic. This method does not work always since we have some instructions which only has rtn or nortn version. One such instruction is ds_wrxchg_rtn_b32 which does not have nortn version. This has caused changes in memory legalizer tests. Differential Revision: https://reviews.llvm.org/D96639
2021-02-14[AMDGPU] Limit memory scope for scratch, LDS and GDSTony Tye
Changes for AMD GPU SIMemoryLegalizer: - Limit the memory scope to maximum supported by the scratch, LDS and GDS address spaces. - Improve assertion checking. - Correct toSIAtomicScope argument name. Reviewed By: rampitec Differential Revision: https://reviews.llvm.org/D96643
generated by cgit v1.2.3 (git 2.25.1) at 2026-10-24 19:10:09 +0000