<feed xmlns='http://www.w3.org/2005/Atom'>
<title>llvm-project.git/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp, branch main</title>
<subtitle>Unnamed repository; edit this file 'description' to name the repository.
</subtitle>
<link rel='alternate' type='text/html' href='https://git.belthelziquor.com/llvm-project.git/'/>
<entry>
<title>[mlir][vector] Missing indices on vectorization of 1-d reduction to 1-ranked memref (#166959)</title>
<updated>2025-11-19T14:52:27+00:00</updated>
<author>
<name>Simone Pellegrini</name>
<email>simone.pellegrini@arm.com</email>
</author>
<published>2025-11-19T14:52:27+00:00</published>
<link rel='alternate' type='text/html' href='https://git.belthelziquor.com/llvm-project.git/commit/?id=71e3de8a7f1c0fc71302ac84c826f34fa324ee1c'/>
<id>71e3de8a7f1c0fc71302ac84c826f34fa324ee1c</id>
<content type='text'>
Vectorization of a 1-d reduction where the output variable is a 1-ranked
memref can generate an invalid `vector.transfer_write` with no indices
for the memref, e.g.:

vector.transfer_write"(%vec, %buff) &lt;{...}&gt; : (vector&lt;f32&gt;,
memref&lt;1xf32&gt;) -&gt; ()

This patch solves the problem by providing the expected amount of
indices (i.e. matching the rank of the memref).</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Vectorization of a 1-d reduction where the output variable is a 1-ranked
memref can generate an invalid `vector.transfer_write` with no indices
for the memref, e.g.:

vector.transfer_write"(%vec, %buff) &lt;{...}&gt; : (vector&lt;f32&gt;,
memref&lt;1xf32&gt;) -&gt; ()

This patch solves the problem by providing the expected amount of
indices (i.e. matching the rank of the memref).</pre>
</div>
</content>
</entry>
<entry>
<title>[mlir][vector] Simplify createReadOrMaskedRead (#163736)</title>
<updated>2025-11-11T17:39:56+00:00</updated>
<author>
<name>Andrzej Warzyński</name>
<email>andrzej.warzynski@arm.com</email>
</author>
<published>2025-11-11T17:39:56+00:00</published>
<link rel='alternate' type='text/html' href='https://git.belthelziquor.com/llvm-project.git/commit/?id=04df359ae1f0209500c4dd4af1167b22e1b75d5f'/>
<id>04df359ae1f0209500c4dd4af1167b22e1b75d5f</id>
<content type='text'>
Simplify `createReadOrMaskedRead` to only require _one_ argument to
specify the vector type to read (passed as `VectorType`) instead of
passing vector-sizes and scalable-flags independently (i.e. _two_
arguments).

A simple overload is provided for users that wouldn't re-use the
corresponding `VectorType` (and hence there's no point for them
to create). While there are no users upstream for this overload,
it's been helpful downstream.</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Simplify `createReadOrMaskedRead` to only require _one_ argument to
specify the vector type to read (passed as `VectorType`) instead of
passing vector-sizes and scalable-flags independently (i.e. _two_
arguments).

A simple overload is provided for users that wouldn't re-use the
corresponding `VectorType` (and hence there's no point for them
to create). While there are no users upstream for this overload,
it's been helpful downstream.</pre>
</div>
</content>
</entry>
<entry>
<title>[mlir][linalg] Update vectorization of linalg.pack (#163539)</title>
<updated>2025-11-06T13:53:56+00:00</updated>
<author>
<name>Andrzej Warzyński</name>
<email>andrzej.warzynski@arm.com</email>
</author>
<published>2025-11-06T13:53:56+00:00</published>
<link rel='alternate' type='text/html' href='https://git.belthelziquor.com/llvm-project.git/commit/?id=75573041969a0d42cbf771c4b3c3ccea661f44aa'/>
<id>75573041969a0d42cbf771c4b3c3ccea661f44aa</id>
<content type='text'>
This patch changes `vectorizeAsTensorPackOp` to require users to specify
**all** write-side vector sizes for `linalg.pack` (not just the outer
dimensions). This makes `linalg.pack` vectorization consistent with
`linalg.unpack` (see https://github.com/llvm/llvm-project/pull/149293
for a similar change).

Conceptually, `linalg.pack` consists of these high-level steps:
  * **Read** from the source tensor using `vector.transfer_read`.
  * **Re-associate** dimensions of the read value, as specified by
    the op (via `vector.shape_cast`)
  * **Transpose** the re-associated value according to the permutation
    in the `linalg.pack` op (via `vector.transpose`).
  * **Write** the result into the destination tensor via
    `vector.transfer_write`.

Previously, the vector sizes provided by the user were interpreted as
write-vector-sizes for PackOp **_outer_** dims (i.e. the final step
above). These were used to:
  * Infer read-vector-sizes using the `inner_tiles` attribute of PackOp.
  * Deduce vector sizes for the transpose and shape cast operations.
  * Ultimately determine the vector shape for the read.

However, this logic breaks when one or more tile sizes are dynamic (*).
In such cases, `vectorizePackOpPrecondition` would currently fail (see
`@pack_with_dynamic_dims_and_dynamic_inner_tile` added in this PR -
without this change it will crash).

This patch updates the contract: users now directly specify _all_ the
"write-vector-sizes", which inherently encode all inner tile sizes -
including dynamic ones. It becomes the user's responsibility to provide
valid sizes.

In practice, since `linalg.pack` is typically constructed, tiled, and
vectorized by the same transformation pipeline, the necessary
"write-vector-sizes" should be recoverable.

Notes for reviewers:
  * See test updates for user-facing impact.
  * Review `vectorizeAsTensorPackOp` as a new implementation rather than
    a diff.
  * Comments and variable names were updated to align with
    `vectorizeAsTensorUnPackOp`.

(*) As a concrete example, "scalable" tile sizes are represent as
dynamic values. Note, support for "scalable" vectorisation will be added
in a separate PR.</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This patch changes `vectorizeAsTensorPackOp` to require users to specify
**all** write-side vector sizes for `linalg.pack` (not just the outer
dimensions). This makes `linalg.pack` vectorization consistent with
`linalg.unpack` (see https://github.com/llvm/llvm-project/pull/149293
for a similar change).

Conceptually, `linalg.pack` consists of these high-level steps:
  * **Read** from the source tensor using `vector.transfer_read`.
  * **Re-associate** dimensions of the read value, as specified by
    the op (via `vector.shape_cast`)
  * **Transpose** the re-associated value according to the permutation
    in the `linalg.pack` op (via `vector.transpose`).
  * **Write** the result into the destination tensor via
    `vector.transfer_write`.

Previously, the vector sizes provided by the user were interpreted as
write-vector-sizes for PackOp **_outer_** dims (i.e. the final step
above). These were used to:
  * Infer read-vector-sizes using the `inner_tiles` attribute of PackOp.
  * Deduce vector sizes for the transpose and shape cast operations.
  * Ultimately determine the vector shape for the read.

However, this logic breaks when one or more tile sizes are dynamic (*).
In such cases, `vectorizePackOpPrecondition` would currently fail (see
`@pack_with_dynamic_dims_and_dynamic_inner_tile` added in this PR -
without this change it will crash).

This patch updates the contract: users now directly specify _all_ the
"write-vector-sizes", which inherently encode all inner tile sizes -
including dynamic ones. It becomes the user's responsibility to provide
valid sizes.

In practice, since `linalg.pack` is typically constructed, tiled, and
vectorized by the same transformation pipeline, the necessary
"write-vector-sizes" should be recoverable.

Notes for reviewers:
  * See test updates for user-facing impact.
  * Review `vectorizeAsTensorPackOp` as a new implementation rather than
    a diff.
  * Comments and variable names were updated to align with
    `vectorizeAsTensorUnPackOp`.

(*) As a concrete example, "scalable" tile sizes are represent as
dynamic values. Note, support for "scalable" vectorisation will be added
in a separate PR.</pre>
</div>
</content>
</entry>
<entry>
<title>[mlir] Simplify Default cases in type switches. NFC. (#165767)</title>
<updated>2025-10-30T19:10:59+00:00</updated>
<author>
<name>Jakub Kuderski</name>
<email>jakub@nod-labs.com</email>
</author>
<published>2025-10-30T19:10:59+00:00</published>
<link rel='alternate' type='text/html' href='https://git.belthelziquor.com/llvm-project.git/commit/?id=ba0be89cd2c08dfff483cbcbdb770dce44031b6c'/>
<id>ba0be89cd2c08dfff483cbcbdb770dce44031b6c</id>
<content type='text'>
Use default values instead of lambdas when possible. `std::nullopt` and
`nullptr` can be used now because of
https://github.com/llvm/llvm-project/pull/165724.</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Use default values instead of lambdas when possible. `std::nullopt` and
`nullptr` can be used now because of
https://github.com/llvm/llvm-project/pull/165724.</pre>
</div>
</content>
</entry>
<entry>
<title>[MLIR] Apply clang-tidy fixes for llvm-qualified-auto in Vectorization.cpp (NFC)</title>
<updated>2025-10-30T06:48:05+00:00</updated>
<author>
<name>Mehdi Amini</name>
<email>joker.eph@gmail.com</email>
</author>
<published>2025-08-21T14:32:24+00:00</published>
<link rel='alternate' type='text/html' href='https://git.belthelziquor.com/llvm-project.git/commit/?id=e0d9c9c33329f5801c832661132a8bef2f4be836'/>
<id>e0d9c9c33329f5801c832661132a8bef2f4be836</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[mlir][linalg] set inbounds on `xfer_read/writes` for `assumeDynamicDimsMatchVecSizes ` (#160839)</title>
<updated>2025-10-09T16:23:18+00:00</updated>
<author>
<name>Ege Beysel</name>
<email>beysel@roofline.ai</email>
</author>
<published>2025-10-09T16:23:18+00:00</published>
<link rel='alternate' type='text/html' href='https://git.belthelziquor.com/llvm-project.git/commit/?id=644b068b35b430b279eb3924ac94c6c974588519'/>
<id>644b068b35b430b279eb3924ac94c6c974588519</id>
<content type='text'>
The idea from #146531 was to introduce the flag
`assumeDynamicDimsMatchVecSizes`, to signal the vectorizer that the
access should not be masked and is in-bounds. Though the masking part is
handled, `xfer_read/write` ops are created without explicitly setting
the inbounds attribute, which defaults to all-false.

In the existence of scalable tile sizes, subsequent patterns tend to
overwrite the inbounds attribute and introduce masks further down when
lowered to loads and stores. This PR explicitly sets the inbounds
attribute to all-true for `xfer_read/write` ops if the
`assumeDynamicDimsMatchVecSizes` flag is set.

---------

Signed-off-by: Ege Beysel &lt;beyselege@gmail.com&gt;</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The idea from #146531 was to introduce the flag
`assumeDynamicDimsMatchVecSizes`, to signal the vectorizer that the
access should not be masked and is in-bounds. Though the masking part is
handled, `xfer_read/write` ops are created without explicitly setting
the inbounds attribute, which defaults to all-false.

In the existence of scalable tile sizes, subsequent patterns tend to
overwrite the inbounds attribute and introduce masks further down when
lowered to loads and stores. This PR explicitly sets the inbounds
attribute to all-true for `xfer_read/write` ops if the
`assumeDynamicDimsMatchVecSizes` flag is set.

---------

Signed-off-by: Ege Beysel &lt;beyselege@gmail.com&gt;</pre>
</div>
</content>
</entry>
<entry>
<title>[MLIR] Apply clang-tidy fixes for readability-container-size-empty in Vectorization.cpp (NFC)</title>
<updated>2025-09-30T08:26:40+00:00</updated>
<author>
<name>Mehdi Amini</name>
<email>joker.eph@gmail.com</email>
</author>
<published>2025-08-21T14:33:15+00:00</published>
<link rel='alternate' type='text/html' href='https://git.belthelziquor.com/llvm-project.git/commit/?id=f2c9abc2be79f4dab0103c79e5582cb6936e253f'/>
<id>f2c9abc2be79f4dab0103c79e5582cb6936e253f</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[mlir][linalg] Use ub.poison when vectorizing pack+unpack Ops (#159536)</title>
<updated>2025-09-23T11:27:48+00:00</updated>
<author>
<name>Andrzej Warzyński</name>
<email>andrzej.warzynski@arm.com</email>
</author>
<published>2025-09-23T11:27:48+00:00</published>
<link rel='alternate' type='text/html' href='https://git.belthelziquor.com/llvm-project.git/commit/?id=862a1bea6fe1ecab3a3fce03314761b16871f2b1'/>
<id>862a1bea6fe1ecab3a3fce03314761b16871f2b1</id>
<content type='text'>
This patch makes sure that in the absence of an explicit pad value in
`linalg.pack`, the vectorizer will use `ub.poison` for the corresponding
Xfer Op pad value (as opposed to e.g. `arith.constant 0`).

Also, in the case of `linalg.unpack`, use `ub.poison` for the Xfer read
operation. In this case, there is no mechanism for a user to specify the
pad/pass-thru value.</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This patch makes sure that in the absence of an explicit pad value in
`linalg.pack`, the vectorizer will use `ub.poison` for the corresponding
Xfer Op pad value (as opposed to e.g. `arith.constant 0`).

Also, in the case of `linalg.unpack`, use `ub.poison` for the Xfer read
operation. In this case, there is no mechanism for a user to specify the
pad/pass-thru value.</pre>
</div>
</content>
</entry>
<entry>
<title>[mlir][linalg] Update vectorization logic for linalg.pack (#149156) (#158926)</title>
<updated>2025-09-18T09:32:46+00:00</updated>
<author>
<name>Andrzej Warzyński</name>
<email>andrzej.warzynski@arm.com</email>
</author>
<published>2025-09-18T09:32:46+00:00</published>
<link rel='alternate' type='text/html' href='https://git.belthelziquor.com/llvm-project.git/commit/?id=2f6b433f2898f6d431dfefd393c2c7777c740418'/>
<id>2f6b433f2898f6d431dfefd393c2c7777c740418</id>
<content type='text'>
NOTE: See #149156 for a smilar change for `linalg.unpack`

This PR makes sure that we don't generate unnecessary `tensor.empty`
when vectorizing `linalg.pack`.

To better visualize the changes implemented here, consider this IR:
```mlir
func.func @example(
    %src: tensor&lt;64x4xf32&gt;,
    %dest: tensor&lt;2x4x16x2xf32&gt;) -&gt; tensor&lt;2x4x16x2xf32&gt; {

  %pack = linalg.pack %src
    outer_dims_perm = [1, 0]
    inner_dims_pos = [0, 1]
    inner_tiles = [16, 2]
    into %dest : tensor&lt;64x4xf32&gt; -&gt; tensor&lt;2x4x16x2xf32&gt;

  return %pack : tensor&lt;2x4x16x2xf32&gt;
}
```

Below is the output after vectorization, BEFORE and AFTER this PR.

BEFORE (note `tensor.empty` and the fact that `%arg1` is not used):
```mlir
  func.func @example(%arg0: tensor&lt;64x4xf32&gt;, %arg1: tensor&lt;2x4x16x2xf32&gt;) -&gt; tensor&lt;2x4x16x2xf32&gt; {
    %cst = arith.constant 0.000000e+00 : f32
    %c0 = arith.constant 0 : index
    %0 = vector.transfer_read %arg0[%c0, %c0], %cst {in_bounds = [true, true]} : tensor&lt;64x4xf32&gt;, vector&lt;64x4xf32&gt;
    %1 = vector.shape_cast %0 : vector&lt;64x4xf32&gt; to vector&lt;4x16x2x2xf32&gt;
    %2 = vector.transpose %1, [2, 0, 1, 3] : vector&lt;4x16x2x2xf32&gt; to vector&lt;2x4x16x2xf32&gt;
    %3 = tensor.empty() : tensor&lt;2x4x16x2xf32&gt;
    %c0_0 = arith.constant 0 : index
    %4 = vector.transfer_write %2, %3[%c0_0, %c0_0, %c0_0, %c0_0] {in_bounds = [true, true, true, true]} : vector&lt;2x4x16x2xf32&gt;, tensor&lt;2x4x16x2xf32&gt;
    return %4 : tensor&lt;2x4x16x2xf32&gt;
  }
```

AFTER (note that `%arg1` is correctly used):
```mlir
func.func @example(%arg0: tensor&lt;64x4xf32&gt;, %arg1: tensor&lt;2x4x16x2xf32&gt;) -&gt; tensor&lt;2x4x16x2xf32&gt; {
  %cst = arith.constant 0.000000e+00 : f32
  %c0 = arith.constant 0 : index
  %0 = vector.transfer_read %arg0[%c0, %c0], %cst {in_bounds = [true, true]} : tensor&lt;64x4xf32&gt;, vector&lt;64x4xf32&gt;
  %1 = vector.shape_cast %0 : vector&lt;64x4xf32&gt; to vector&lt;4x16x2x2xf32&gt;
  %2 = vector.transpose %1, [2, 0, 1, 3] : vector&lt;4x16x2x2xf32&gt; to vector&lt;2x4x16x2xf32&gt;
  %c0_0 = arith.constant 0 : index
  %3 = vector.transfer_write %2, %arg1[%c0_0, %c0_0, %c0_0, %c0_0] {in_bounds = [true, true, true, true]} : vector&lt;2x4x16x2xf32&gt;, tensor&lt;2x4x16x2xf32&gt;
  return %3 : tensor&lt;2x4x16x2xf32&gt;
}
```

ADDITIONAL CHANGES:
  * Adds missing `CHECK-LABEL` in tests.
  * Capitalize LIT test variables names.</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
NOTE: See #149156 for a smilar change for `linalg.unpack`

This PR makes sure that we don't generate unnecessary `tensor.empty`
when vectorizing `linalg.pack`.

To better visualize the changes implemented here, consider this IR:
```mlir
func.func @example(
    %src: tensor&lt;64x4xf32&gt;,
    %dest: tensor&lt;2x4x16x2xf32&gt;) -&gt; tensor&lt;2x4x16x2xf32&gt; {

  %pack = linalg.pack %src
    outer_dims_perm = [1, 0]
    inner_dims_pos = [0, 1]
    inner_tiles = [16, 2]
    into %dest : tensor&lt;64x4xf32&gt; -&gt; tensor&lt;2x4x16x2xf32&gt;

  return %pack : tensor&lt;2x4x16x2xf32&gt;
}
```

Below is the output after vectorization, BEFORE and AFTER this PR.

BEFORE (note `tensor.empty` and the fact that `%arg1` is not used):
```mlir
  func.func @example(%arg0: tensor&lt;64x4xf32&gt;, %arg1: tensor&lt;2x4x16x2xf32&gt;) -&gt; tensor&lt;2x4x16x2xf32&gt; {
    %cst = arith.constant 0.000000e+00 : f32
    %c0 = arith.constant 0 : index
    %0 = vector.transfer_read %arg0[%c0, %c0], %cst {in_bounds = [true, true]} : tensor&lt;64x4xf32&gt;, vector&lt;64x4xf32&gt;
    %1 = vector.shape_cast %0 : vector&lt;64x4xf32&gt; to vector&lt;4x16x2x2xf32&gt;
    %2 = vector.transpose %1, [2, 0, 1, 3] : vector&lt;4x16x2x2xf32&gt; to vector&lt;2x4x16x2xf32&gt;
    %3 = tensor.empty() : tensor&lt;2x4x16x2xf32&gt;
    %c0_0 = arith.constant 0 : index
    %4 = vector.transfer_write %2, %3[%c0_0, %c0_0, %c0_0, %c0_0] {in_bounds = [true, true, true, true]} : vector&lt;2x4x16x2xf32&gt;, tensor&lt;2x4x16x2xf32&gt;
    return %4 : tensor&lt;2x4x16x2xf32&gt;
  }
```

AFTER (note that `%arg1` is correctly used):
```mlir
func.func @example(%arg0: tensor&lt;64x4xf32&gt;, %arg1: tensor&lt;2x4x16x2xf32&gt;) -&gt; tensor&lt;2x4x16x2xf32&gt; {
  %cst = arith.constant 0.000000e+00 : f32
  %c0 = arith.constant 0 : index
  %0 = vector.transfer_read %arg0[%c0, %c0], %cst {in_bounds = [true, true]} : tensor&lt;64x4xf32&gt;, vector&lt;64x4xf32&gt;
  %1 = vector.shape_cast %0 : vector&lt;64x4xf32&gt; to vector&lt;4x16x2x2xf32&gt;
  %2 = vector.transpose %1, [2, 0, 1, 3] : vector&lt;4x16x2x2xf32&gt; to vector&lt;2x4x16x2xf32&gt;
  %c0_0 = arith.constant 0 : index
  %3 = vector.transfer_write %2, %arg1[%c0_0, %c0_0, %c0_0, %c0_0] {in_bounds = [true, true, true, true]} : vector&lt;2x4x16x2xf32&gt;, tensor&lt;2x4x16x2xf32&gt;
  return %3 : tensor&lt;2x4x16x2xf32&gt;
}
```

ADDITIONAL CHANGES:
  * Adds missing `CHECK-LABEL` in tests.
  * Capitalize LIT test variables names.</pre>
</div>
</content>
</entry>
<entry>
<title>[mlir][linalg] Add support for scalable vectorization of `linalg.batch_mmt4d` (#152984)</title>
<updated>2025-08-14T09:47:51+00:00</updated>
<author>
<name>Ege Beysel</name>
<email>beyselege@gmail.com</email>
</author>
<published>2025-08-14T09:47:51+00:00</published>
<link rel='alternate' type='text/html' href='https://git.belthelziquor.com/llvm-project.git/commit/?id=8de85e753fc995d1a8b980c35dc9413ae51d0a1a'/>
<id>8de85e753fc995d1a8b980c35dc9413ae51d0a1a</id>
<content type='text'>
This PR builds upon the previous #146531 and enables scalable
vectorization for `batch_mmt4d` as well.

---------

Signed-off-by: Ege Beysel &lt;beyselege@gmail.com&gt;</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR builds upon the previous #146531 and enables scalable
vectorization for `batch_mmt4d` as well.

---------

Signed-off-by: Ege Beysel &lt;beyselege@gmail.com&gt;</pre>
</div>
</content>
</entry>
</feed>
