| Age | Commit message (Collapse) | Author |
|---|
|
This PR makes the following improvements to `vector.scatter` and its
lowering pipeline:
- In addition to `memref`, accept a ranked `tensor` as the base operand
of `vector.scatter`, similar to `vector.transfer_write`.
- Implement bufferization support for `vector.scatter`, so that
tensor-based scatter ops can be fully lowered to memref-based forms.
It's worth to complete the functionality of map_scatter decomposition.
Full discussion can be found here:
https://github.com/iree-org/iree/issues/21135
---------
Signed-off-by: Ryutaro Okada <1015ryu88@gmail.com>
|
|
(#167848)
Reland pass and fix linker errors.
---------
Co-authored-by: Maksim Levental <maksim.levental@gmail.com>
|
|
pass: `arith-to-apfloat`" (#167834)
Reverts llvm/llvm-project#167608
Broken builder https://lab.llvm.org/buildbot/#/builders/52/builds/12781
|
|
`arith-to-apfloat` (#167608)
Fix both symbol visibility issue in the mlir_apfloat_wrappers lib and the linkage issue in ArithToAPFloat.
|
|
pass: `arith-to-apfloat` (#166618)" (#167431)"" (#167549)
Reverts llvm/llvm-project#167436 to fix sanitizers
|
|
`arith-to-apfloat` (#166618)" (#167431)" (#167436)
Reland https://github.com/llvm/llvm-project/pull/166618 by fixing
missing symbol issues by explicitly loading
`--shared-libs=%mlir_apfloat_wrappers` as well as
`--shared-libs=%mlir_c_runner_utils`.
|
|
`arith-to-apfloat` (#166618)" (#167431)" (#167435)
This reverts commit 0e639ae6e030ade849fa7a09cb7dc40b42f25373.
|
|
`arith-to-apfloat` (#166618)" (#167431)
Reland https://github.com/llvm/llvm-project/pull/166618 with
MLIRFuncUtils linked in.
|
|
`arith-to-apfloat` (#166618)" (#167429)
This reverts commit 222f4e494a0cd9515c242fd083c2776772734385.
|
|
(#166618)
This commit adds a new pass that lowers floating-point `arith`
operations to calls into the execution engine runtime library. Currently
supported operations: `addf`, `subf`, `mulf`, `divf`, `remf`.
All floating-point types that have an APFloat semantics are supported.
This includes low-precision floating-point types such as `f4E2M1FN` that
cannot execute natively on CPUs.
This commit also improves the `vector.print` lowering pattern to call
into the runtime library for floating-point types that are not supported
by LLVM. This is necessary to write a meaningful integration test.
The way it works is
```mlir
func.func @full_example() {
%a = arith.constant 1.4 : f8E4M3FN
%b = func.call @foo() : () -> (f8E4M3FN)
%c = arith.addf %a, %b : f8E4M3FN
vector.print %c : f8E4M3FN
return
}
```
gets transformed to
```mlir
func.func private @__mlir_apfloat_add(i32, i64, i64) -> i6
func.func @full_example() {
%cst = arith.constant 1.375000e+00 : f8E4M3FN
%0 = call @foo() : () -> f8E4M3FN
// bitcast operand A to integer of equal width
%1 = arith.bitcast %cst : f8E4M3FN to i8
// zext A to i64
%2 = arith.extui %1 : i8 to i64
// same for operand B
%3 = arith.bitcast %0 : f8E4M3FN to i8
%4 = arith.extui %3 : i8 to i64
// get the llvm::fltSemantics(f8E4M3FN) as an enum
%c10_i32 = arith.constant 10 : i32
// call the impl against APFloat in mlir_apfloat_wrappers
%5 = call @__mlir_apfloat_add(%c10_i32, %2, %4) : (i32, i64, i64) -> i64
// "cast" back to the original fp type
%6 = arith.trunci %5 : i64 to i8
%7 = arith.bitcast %6 : i8 to f8E4M3FN
vector.print %7 : f8E4M3FN
}
```
Note, `llvm::fltSemantics(f8E4M3FN)` is emitted by the pattern each time
an `arith` op is transformed, thereby making the call to
`__mlir_apfloat_add` correct (i.e., no name mangling on type necessary).
RFC:
https://discourse.llvm.org/t/rfc-software-implementation-for-unsupported-fp-types-in-convert-arith-to-llvm/88785
---------
Co-authored-by: Matthias Springer <me@m-sp.org>
|
|
ConvertVectorToLLVM.cpp (NFC)
|
|
if vector.broadcast source is a scalar and target is a single element 1D
vector.
|
|
vector.splat has been deprecated (user: please use the very similar vector.broadcast instead)
with the last PR landing about 6 weeks ago.
The discourse discussion is at
https://discourse.llvm.org/t/rfc-mlir-vector-deprecate-then-remove-vector-splat/87143/1
The last PR was #152230
This PR completely removes vector.splat. In addition to removing vector.splat from VectorOps.td, it
- Updates the few remaining places where vector::SplatOp is created (now vector::BroadcastOp is created)
- Removes temporary patterns where vector.splat is replaced by vector.broadcast
The only place 'vector.splat' appears is now the files
https://github.com/llvm/llvm-project/blob/main/mlir/utils/tree-sitter-mlir/test/corpus/op.txt
and
https://github.com/llvm/llvm-project/blob/main/mlir/utils/tree-sitter-mlir/dialect/vector.js
---------
Signed-off-by: James Newling <james.newling@gmail.com>
|
|
Use the `Base` type alias from
https://github.com/llvm/llvm-project/pull/158433.
|
|
This is a follow-up to https://github.com/llvm/llvm-project/pull/144307,
where we removed `vector.matrix_multiply` and `vector.flat_transpose`
from the Vector dialect.
This PR:
* Updates comments that were missed in the previous change.
* Renames relevant `-convert-vector-to-llvm=` options:
- `vector-contract-lowering=matmul` → `vector-contract-lowering=llvmintr`
- `vector-transpose-lowering=flat_transpose` → `vector-transpose-lowering=llvmintr`
These new names better reflect the actual transformation target - LLVM
intrinsics - rather than the now-removed abstract operations.
|
|
This patch updates the following ops to use `source` (instead of
`vector`) as the name for their source argument:
* `vector.extract`
* `vector.scalable.extract`
* `vector.extract_strided_slice`
This change ensures naming consistency with the "builders" for these Ops
that already use the name `source` rather than `vector`. It also
addresses part of:
* https://github.com/llvm/llvm-project/issues/131602
Specifically, it ensures that we use `source` and `dest` for read and
write operations, respectively (as opposed to `vector` and `dest`).
|
|
ConvertVectorToLLVM.cpp (NFC)
|
|
Allows alignment to be propagated correctly from vector to LLVM dialect
operations.
|
|
Renames `indices` as `offsets` and `index_vec` as `indices`. This is
primarily to make clearer distinction between the arguments.
|
|
VectorFromElementsLowering (#151175)
This patch introduces a new unrolling-based approach for lowering
multi-dimensional `vector.from_elements` operations.
**Implementation Details:**
1. **New Transform Pattern**: Added `UnrollFromElements` that unrolls a
N-D(N>=2) from_elements op to a (N-1)-D from_elements op align the
outermost dimension.
2. **Utility Functions**: Added `unrollVectorOp` to reuse the unroll
algo of vector.gather for vector.from_elements.
3. **Integration**: Added the unrolling pattern to the
convert-vector-to-llvm pass as a temporal transformation.
4. Use direct LLVM dialect operations instead of intermediate
vector.insert operations for efficiency in `VectorFromElementsLowering`.
**Example:**
```mlir
// unroll
%v = vector.from_elements %e0, %e1, %e2, %e3 : vector<2x2xf32>
=>
%poison_2d = ub.poison : vector<2x2xf32>
%vec_1d_0 = vector.from_elements %e0, %e1 : vector<2xf32>
%vec_2d_0 = vector.insert %vec_1d_0, %poison_2d [0] : vector<2xf32> into vector<2x2xf32>
%vec_1d_1 = vector.from_elements %e2, %e3 : vector<2xf32>
%result = vector.insert %vec_1d_1, %vec_2d_0 [1] : vector<2xf32> into vector<2x2xf32>
// convert-vector-to-llvm
%v = vector.from_elements %e0, %e1, %e2, %e3 : vector<2x2xf32>
=>
%poison_2d = ub.poison : vector<2x2xf32>
%poison_2d_cast = builtin.unrealized_conversion_cast %poison_2d : vector<2x2xf32> to !llvm.array<2 x vector<2xf32>>
%poison_1d_0 = llvm.mlir.poison : vector<2xf32>
%c0_0 = llvm.mlir.constant(0 : i64) : i64
%vec_1d_0_0 = llvm.insertelement %e0, %poison_1d_0[%c0_0 : i64] : vector<2xf32>
%c1_0 = llvm.mlir.constant(1 : i64) : i64
%vec_1d_0_1 = llvm.insertelement %e1, %vec_1d_0_0[%c1_0 : i64] : vector<2xf32>
%vec_2d_0 = llvm.insertvalue %vec_1d_0_1, %poison_2d_cast[0] : !llvm.array<2 x vector<2xf32>>
%poison_1d_1 = llvm.mlir.poison : vector<2xf32>
%c0_1 = llvm.mlir.constant(0 : i64) : i64
%vec_1d_1_0 = llvm.insertelement %e2, %poison_1d_1[%c0_1 : i64] : vector<2xf32>
%c1_1 = llvm.mlir.constant(1 : i64) : i64
%vec_1d_1_1 = llvm.insertelement %e3, %vec_1d_1_0[%c1_1 : i64] : vector<2xf32>
%vec_2d_1 = llvm.insertvalue %vec_1d_1_1, %vec_2d_0[1] : !llvm.array<2 x vector<2xf32>>
%result = builtin.unrealized_conversion_cast %vec_2d_1 : !llvm.array<2 x vector<2xf32>> to vector<2x2xf32>
```
---------
Co-authored-by: Nicolas Vasilache <Nico.Vasilache@amd.com>
Co-authored-by: Yang Bai <yangb@nvidia.com>
Co-authored-by: James Newling <james.newling@gmail.com>
Co-authored-by: Diego Caballero <dieg0ca6aller0@gmail.com>
|
|
(#151690)
These commits continue the work done in
https://github.com/llvm/llvm-project/pull/144344, of adding alignment
attributes to operations in the vector and memref. These commits focus
on adding the alignment attribute to the `maskedload` and `maskedstore`
operations. The `VectorLoadConversion` pattern in VectorToLLVM is a
template for `load`, `store`, `maskedload` and `maskedstore` operations.
Having the alignment attribute in all these operations would allow for
an easy way to propagate the alignment attribute from the vector dialect
to the LLVM dialect.
This patchset also includes changes to the conversion from VectorToLLVM
to propagate the alignment attribute for the
vector.{,masked}{load,store} operations.
|
|
(#149603)
This PR removes `vector.extractelement` and `vector.insertelement` ops
from the code base in favor of the `vector.extract` and `vector.insert`
counterparts.
See RFC:
https://discourse.llvm.org/t/rfc-psa-remove-vector-extractelement-and-vector-insertelement-ops-in-favor-of-vector-extract-and-vector-insert-ops
|
|
This patch removes spurious includes of `llvm/IR` files, and unnecessary
link components in the LLVM dialect.
The only major dependencies still coming from LLVM are
`llvm::DataLayout`, which is used by `verifyDataLayoutString` and some
`dwarf` symbols in some attributes. Both of them should likely be
removed in the future.
Finally, I also removed one constructor from `LLVM::AssumeOp` that used
[OperandBundleDefT](https://llvm.org/doxygen/classllvm_1_1OperandBundleDefT.html)
without good reason and introduced a header unnecessarily.
|
|
See https://github.com/llvm/llvm-project/pull/147168 for more info.
|
|
Add conversion for broadcast from scalar for LLVM and SPIRV. Also some
miscellaneous replacements of vector.splat with vector.broadcast in
VectorToGPU and ArithToAMDGPU.
Part of deprecation of vector.splat RFC:
https://discourse.llvm.org/t/rfc-mlir-vector-deprecate-then-remove-vector-splat/87143/4
|
|
dialect (#144307)
This patch deletes `vector.matrix_multiply` and `vector.flat_transpose`,
which are thin wrappers around the corresponding LLVM intrinsics:
- `llvm.intr.matrix.multiply`
- `llvm.intr.matrix.transpose`
These Vector dialect ops did not provide additional semantics or
abstraction beyond the LLVM intrinsics. Their removal simplifies the
lowering pipeline without losing any functionality.
The lowering chains:
- `vector.contract` → `vector.matrix_multiply` →
`llvm.intr.matrix.multiply`
- `vector.transpose` → `vector.flat_transpose` →
`llvm.intr.matrix.transpose`
are now replaced with:
- `vector.contract` → `llvm.intr.matrix.multiply`
- `vector.transpose` → `llvm.intr.matrix.transpose`
This was accomplished by directly replacing:
- `vector::MatrixMultiplyOp` with `LLVM::MatrixMultiplyOp`
- `vector::FlatTransposeOp` with `LLVM::MatrixTransposeOp`
Note: To avoid a build-time dependency from `Vector` to `LLVM`,
relevant transformations are moved from "Vector/Transforms" to
`Conversion/VectorToLLVM`.
|
|
Only elements with at least one use are lowered to `llvm.extractelement`
op.
|
|
This PR allows to optionally speed up the lookup of symbols by providing a `SymbolTableCollection` instance to the interested conversion patterns. It is follow-up on the discussion about symbol / symbol table management carried on [Discourse](https://discourse.llvm.org/t/symbol-table-as-first-class-citizen-in-builders/86813).
|
|
Now that MLIR accepts nuw and nusw in getelementptr, this patch emits
the inbounds and nuw attributes when lower memref to LLVM in load and
store operators.
This patch also strengthens the memref.load and memref.store spec about
undefined behaviour during lowering.
This patch also lifts the |rewriter| parameter in getStridedElementPtr
ahead so that LLVM::GEPNoWrapFlags can be added at the end with a
default value and grouped together with other operators' parameters.
Signed-off-by: Lin, Peiyong <linpyong@gmail.com>
|
|
|
|
In ConvertVectorToLLVM, the only option for setting alignment of
`vector.gather`, `vector.scatter`, and the `vector.load/store` ops was
to extract it from the datatype of the memref type. However, this is
insufficient for hardware backends requiring alignment of vector types.
This PR introduces the `use-vector-alignment` option to the
`ConvertVectorToLLVMPass`, which makes the pass use the alignment of the
vector type of these operations instead of the alignment of the memref
type.
---------
Co-authored-by: Lily Orth-Smith <lorthsmith@microsoft.com>
|
|
These functions are called from lowering patterns. All IR modifications
in a pattern must be performed through the provided rewriter, but these
functions used to instantiate a new `OpBuilder`, bypassing the provided
rewriter.
|
|
These have been deprecated for over two years now in favor of free
functions.
See the relevant discourse thread:
https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443
and the deprecation notice: https://mlir.llvm.org/deprecation/.
|
|
(#134206)
This change standardises the naming convention for the argument
representing the value to store in various vector operations.
Specifically, it ensures that all vector ops storing a value—whether
into memory, a tensor, or another vector — use `valueToStore` for the
corresponding argument name.
Updated operations:
* `vector.transfer_write`, `vector.insert`, `vector.scalable_insert`,
`vector.insert_strided_slice`.
For reference, here are operations that currently use `valueToStore`:
* `vector.store` `vector.scatter`, `vector.compressstore`,
`vector.maskedstore`.
This change is non-functional (NFC) and does not affect the
functionality of these operations.
Implements #131602
|
|
This patch matches the definition of vector.scatter as a counter part of
vector.gather.
All of the changes done in this patch make vector.scatter match
vector.gather 's multi dimensional definition.
Unrolling for vector.scatter will be implemented in subsequent patches.
Discourse Discussion:
https://discourse.llvm.org/t/rfc-improving-gather-codegen-for-vector-dialect/85011/13
|
|
This patch decouples unrolling vector.gather and lowering vector.gather
to llvm.masked.gather.
This is consistent with how vector.load, vector.store,
vector.maskedload, vector.maskedstore lower to LLVM.
Some interesting test changes from this patch:
- 2D vector.gather lowering to llvm tests are deleted. This is
consistent with other memory load/store ops.
- There are still tests for 2D vector.gather, but the constant mask for
these test is modified. This is because with the updated lowering, one
of the unrolled vector.gather disappears because it is masked off (also
demonstrating why this is a better lowering path)
Overall, this makes vector.gather take the same consistent path for
lowering to LLVM as other load/store ops.
Discourse Discussion:
https://discourse.llvm.org/t/rfc-improving-gather-codegen-for-vector-dialect/85011/13
|
|
with `vector.extract` (#128915)
This is doing the same as
https://github.com/llvm/llvm-project/pull/117731 did for
`vector.extract`, but for `vector.insert`.
It is a bit more complicated as the insertion destination may itself
need to be extracted.
As the test shows, this fixes two previously unsupported cases:
- Dynamic indices
- 0-D vectors.
---------
Signed-off-by: Benoit Jacob <jacob.benoit.1@gmail.com>
|
|
LLVM itself is generally moving away from using `undef` and towards
using `poison`, to the point of having a lint that caches new uses of
`undef` in tests.
In order to not trip the lint on new patterns and to conform to the
evolution of LLVM
- Rename valious ::undef() methods on StructBuilder subclasses to
::poison()
- Audit the uses of UndefOp in the MLIR libraries and replace almost all
of them with PoisonOp
The remaining uses of `undef` are initializing `uninitialized` memrefs,
explicit conversions to undef from SPIR-V, and a few cases in
AMDGPUToROCDL where usage like
%v = insertelement <M x iN> undef, iN %v, i32 0
%arg = bitcast <M x iN> %v to i(M * N)
is used to handle "i32" arguments that are are really packed vectors of
smaller types that won't always be fully initialized.
|
|
Create `VectorToLLVMDialectInterface` which allows automatic conversion
discovery by generic `--convert-to-llvm` pass. This only covers final
dialect conversion step and not any previous preparation steps. Also,
currently there is no way to pass any additional parameters through this
conversion interface, but most users using default parameters anyway.
|
|
prototypes (#123378)
On lowering from `memref` to LLVM, `malloc` and other intrinsic
functions from `libc` will be declared in the current module. User's
redefinition of these reserved functions will poison the internal
analysis with wrong prototype. This patch adds assertion on the found
function's type and reports if it mismatch with the intended type.
Related to #120950
---------
Co-authored-by: Luohao Wang <Luohaothu@users.noreply.github.com>
|
|
Turn free-standing `MemRefType`-related helper functions in
`BuiltinTypes.h` into member functions.
|
|
Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:
// FIXME: Replace the uses of is(), get() and dyn_cast() with
// isa<T>, cast<T> and the llvm::dyn_cast<T>
I'm not touching PointerUnion::dyn_cast for now because it's a bit
complicated; we could blindly migrate it to dyn_cast_if_present, but
we should probably use dyn_cast when the operand is known to be
non-null.
|
|
Clean up `populateVectorToLLVMConversionPatterns` so that it populates
only conversion patterns. All rewrite patterns that do not lower to LLVM
should be populated into a separate greedy pattern rewrite.
The current combination of rewrite patterns and conversion patterns
triggered an edge case when merging the 1:1 and 1:N dialect conversions.
Depends on #119973.
|
|
The current implementation of lowering to llvm for vector.extract
incorrectly assumes that if the number of indices is zero, the operation
can be folded away. This PR removes this condition and relies on the
folder to do it instead.
This PR also unifies the logic for scalar extracts and slice extracts,
which as a side effect also enables vector.extract lowering for n-d
vector.extract with dynamic inner most dimension. (This was only
prevented by a conservative check in the old implementation)
|
|
Currently, the lowering for vector.step lives
under a folder. This is not ideal if we want
to do transformation on it and defer the
materizaliztion of the constants much later.
This commits adds a rewrite pattern that
could be used by using
`transform.structured.vectorize_children_and_apply_patterns`
transform dialect operation.
Moreover, the rewriter of vector.step is also
now used in -convert-vector-to-llvm pass where
it handles scalable and non-scalable types as
LLVM expects it.
As a consequence of removing the vector.step
lowering as its folder, linalg vectorization
will keep vector.step intact.
|
|
|
|
This commit marks the type converter in `populate...` functions as
`const`. This is useful for debugging.
Patterns already take a `const` type converter. However, some
`populate...` functions do not only add new patterns, but also add
additional type conversion rules. That makes it difficult to find the
place where a type conversion was added in the code base. With this
change, all `populate...` functions that only populate pattern now have
a `const` type converter. Programmers can then conclude from the
function signature that these functions do not register any new type
conversion rules.
Also some minor cleanups around the 1:N dialect conversion
infrastructure, which did not always pass the type converter as a
`const` object internally.
|
|
VectorExtractOpConversion (#107549)
This patch adds support for converting `vector.extract` that extract
1-element vectors into LLVM, fixing a crash in such cases.
E.g., `vector.extract %1[0]: vector<1xf32> from vector<2xf32>`. Fix
#61372.
|
|
Follow on from #100997. This again removes from boilerplate conversions
to/from IntegerAttr and int64_t (otherwise, this is a NFC).
|
|
In convert-vector-to-llvm the first operand (vector of pointers holding
all memory addresses to read) to the masked.gather (and scatter)
intrinsic has a fixed vector type.
This may result in intrinsics where the scalable flag has been dropped:
```
%0 = llvm.intr.masked.gather %1, %2, %3 {alignment = 4 : i32}
: (!llvm.vec<4 x ptr>, vector<[4]xi1>, vector<[4]xi32>) -> vector<[4]xi32>
```
Fortunately the operand is overloaded on the result type so we end up
with the correct IR when lowering to LLVM, but this is still incorrect.
This patch fixes it by propagating scalability.
|
