| Age | Commit message (Collapse) | Author |
|---|
|
We recently moved over to compiling with clang-cl on Windows. This ended
up causing a large increase in warnings, particularly due to how
warnings are handled in nanobind. cd91d0fff9293a904704784c92c28637bfebef45
initially set -Wall -Wextra and -Wpedantic while fixing another issue,
which is probably not what we want to do on third-party code. We also
need to disable -Wmissing-field-initializers to get things clean in this
configuration.
Reviewers: makslevental, jpienaar, rkayaith
Reviewed By: makslevental
Pull Request: https://github.com/llvm/llvm-project/pull/166828
|
|
pybind (#161230)" (#162309)
This reverts commit 84a214856ad989f37af19f5e8aaa9ec2346dde6f.
This gives us more time to work out the alternative and also people to
migrate
|
|
(#161230)
Inspired by this comment
https://github.com/llvm/llvm-project/pull/157930#issuecomment-3346634290
(and long-standing issues related to finding nanobind/pybind in the
right place), this PR moves to using `FetchContent_Declare` to get the
nanobind dependency. This is pretty standard (see e.g.,
[IREE](https://github.com/iree-org/iree/blob/cf60359b7443b0e92e15fb6ffc011525dc40e772/CMakeLists.txt#L842-L848)).
This PR also removes pybind which has been deprecated for almost a year
(https://github.com/llvm/llvm-project/pull/117922) and which isn't
compatible (for whatever reason) with `FetchContent_Declare`.
---------
Co-authored-by: Jacques Pienaar <jpienaar@google.com>
|
|
If `PYTHONPATH` is set and points to the build location of the python
bindings package then when stubgen runs, `_mlir` will get imported twice
and bad things will happen (e.g., `Assertion !instance && “PyGlobals
already constructed”’`). This happens because `mlir` is a namespace
package and the importer/loader can't distinguish between
`mlir._mlir_libs._mlir` and `_mlir_libs._mlir` imported from `CWD`. Or
something like that. The fix is to filter out any entries in
`PYTHONPATH` that end in `MLIR_BINDINGS_PYTHON_INSTALL_PREFIX/..` (e.g.,
`python_packages/mlir_core/`).
|
|
Silence all stdout from stubgen unless `VERBOSE` is passed to
`mlir_generate_type_stubs`.
|
|
This a reland of https://github.com/llvm/llvm-project/pull/155741 which
was reverted at https://github.com/llvm/llvm-project/pull/157831. This
version is narrower in scope - it only turns on automatic stub
generation for `MLIRPythonExtension.Core._mlir` and **does not do
anything automatically**. Specifically, the only CMake code added to
`AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which
is then used only in a manual way. The API for
`mlir_generate_type_stubs` is:
```
Arguments:
MODULE_NAME: The fully-qualified name of the extension module (used for importing in python).
DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the
specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso)
and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso).
OUTPUT_DIR: The root output directory to emit the type stubs into.
OUTPUTS: List of expected outputs.
DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE).
IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen.
PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files).
Outputs:
NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on.
```
Downstream users should use `mlir_generate_type_stubs` in coordination
with `declare_mlir_python_sources` to turn on stub generation for their
own downstream dialect extensions and upstream dialect extensions if
they so choose. Standalone example shows an example.
Note, downstream will also need to set
`-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
|
|
Despite several hotfixes, things remain broken, in particular:
- installation/distribution (`ninja install / install-distribution`);
- downstream projects with bindings exposed.
See
https://github.com/llvm/llvm-project/pull/157583#issuecomment-3274164357
for more details.
Reverts #155741, #157583, #157697. Let's make sure things are fixed and
re-land as a unit.
|
|
This PR moves generated type stubs to build directory and also quiets
stdout output.
|
|
In https://github.com/llvm/llvm-project/pull/155741 I broke the cardinal
rule of CMake: nothing happens when you think it happens 🤷. Meaning:
`declare_mlir_python_sources(SOURCES_GLOB
"_mlir_libs/${_module_name}/**/*.pyi")` wasn't picking up any sources
_because they aren't generated yet_. This of course makes sense in
retrospect (the stubs are generated as part of the build process, post
extension compile, rather than the configure process).
Thus, the API needs to be:
```
GENERATE_TYPE_STUBS: List of generated type stubs expected from stubgen, relative to _mlir_libs.
```
Partially as a result of this omission, the stubs weren't being
installed into either the build dir nor the install dir. That is also
fixed now:
**Source dir (for easy reference):**
<img width="300" height="674" alt="image"
src="https://github.com/user-attachments/assets/a569f066-c2bd-4361-91f3-1c75181e51da"
/>
**Build dir (for forthcoming typechecker tests):**
<img width="398" height="551" alt="image"
src="https://github.com/user-attachments/assets/017859f9-fddb-49ee-85e5-915f5b5f7377"
/>
**Install dir:**
<img width="456" height="884" alt="image"
src="https://github.com/user-attachments/assets/8051be7e-898c-4ec8-a11e-e2408b241a56"
/>
|
|
This PR turns on automatic type stub generation (rather than relying on
hand-written/updated stubs). It uses nanobind's [stubgen
facility](https://nanobind.readthedocs.io/en/latest/typing.html#stub-generation).
If you would like to enable this functionality you can add
`GENERATE_TYPE_STUBS` to `declare_mlir_python_extension` .
|
|
Introduced (but omitted from this CMake) in
https://github.com/llvm/llvm-project/pull/151246.
|
|
warnings (NFC) (#143863)
Following approach suggested by @hpkfft.
|
|
Set the "-U,_PyClassMethod_New" linker flag on the Python library itself
isntead of on the CMAKE_MODULE_LINKER_FLAGS property.
|
|
Context:
https://github.com/llvm/llvm-project/pull/107103#discussion_r1925834532
This code is brittle, especially when called from a superproject that
adds the `nanobind-*` target in a different source directory:
```cmake
get_property(all_targets DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY BUILDSYSTEM_TARGETS)
```
The changes here do help with my downstream build, but I'm not sure if
using the `MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES` option introduced
in https://github.com/llvm/llvm-project/pull/117934 is the right fix
given that the option is currently scoped directly to one location with
a matching name:
https://github.com/llvm/llvm-project/blob/7ad8a3da4771ce8abbd146611124104d42a4e63e/mlir/cmake/modules/MLIRDetectPythonEnv.cmake#L4-L5
Some other solutions to consider:
1. Search through an explicit list of target names using `if (TARGET)`
2. Iterate over _all_ targets in the project, not just the targets in
the current directory, using code like
https://stackoverflow.com/a/62311397
3. Iterate over targets in the directory known to MLIR
(`llvm-project/mlir/python`)
4. Move this `target_compile_options` setup into
`mlir_configure_python_dev_packages` (I started on this, but that runs
into similar issues where the target is defined in a different
directory)
|
|
See https://cmake.org/cmake/help/latest/policy/CMP0175.html
> The `OUTPUT` form does not accept `PRE_BUILD`, `PRE_LINK`, or
`POST_BUILD` keywords.
When using CMake version 3.31+, this results in ~2000 lines of warning
spam in my downstream project:
```
CMake Warning (dev) at build-gcc/lib/cmake/mlir/AddMLIRPython.cmake:606 (add_custom_command):
The following keywords are not supported when using
add_custom_command(OUTPUT): PRE_BUILD.
Policy CMP0175 is not set: add_custom_command() rejects invalid arguments.
Run "cmake --help-policy CMP0175" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.
Call Stack (most recent call first):
build-gcc/lib/cmake/mlir/AddMLIRPython.cmake:222 (add_mlir_python_sources_target)
build-gcc/lib/cmake/mlir/AddMLIRPython.cmake:256 (_process_target)
compiler/bindings/python/CMakeLists.txt:239 (add_mlir_python_modules)
This warning is for project developers. Use -Wno-dev to suppress it.
```
General docs:
https://cmake.org/cmake/help/latest/command/add_custom_command.html.
Note that `PRE_BUILD` only appears in the _second_ signature for the
function (which takes `TARGET`) not the first (which takes `OUTPUT`).
|
|
`add_mlir_python_extension` (#122865)
This PR allows the users to specify the `NB_DOMAIN` for
`add_mlir_python_extension`. This allows users to avoid nanobind domain
conflicts, when python bindings from multiple `mlir` projects were
imported.
(https://nanobind.readthedocs.io/en/latest/faq.html#how-can-i-avoid-conflicts-with-other-projects-using-nanobind)
|
|
Reland reverted https://github.com/llvm/llvm-project/pull/107103 with
the fixes for Python 3.8
cc @jpienaar
Co-authored-by: Peter Hawkins <phawkins@google.com>
|
|
(#107103)"
Breaks on 3.8, rolling back to avoid breakage while fixing.
This reverts commit 9dee7c44491635ec9037b90050bcdbd3d5291e38.
|
|
Related to https://github.com/llvm/llvm-project/issues/105522
Description:
This PR is a joint work with Peter Hawkins (@hawkinsp) originally done
by myself for pybind11 and then reworked to nanobind based on Peter's
branch: https://github.com/hawkinsp/llvm-project/tree/nbdev .
- Added free-threading CPython mode support for MLIR Python bindings
- Added a test which can reveal data races when cpython and LLVM/MLIR
compiled with TSAN
Context:
- Related to https://github.com/google/jax/issues/23073
Co-authored-by: Peter Hawkins <phawkins@google.com>
|
|
`NanobindAdaptors.h` uses `PyClassMethod_New` to build `pure_subclass`es
but nanobind doesn't declare this API as undefined in its linker flags.
So we need to declare it as such for downstream users that do not do
something like `-undefined dynamic_lookup`
|
|
|
|
This is a companion to #118583, although it can be landed independently
because since #117922 dialects do not have to use the same Python
binding framework as the Python core code.
This PR ports all of the in-tree dialect and pass extensions to
nanobind, with the exception of those that remain for testing pybind11
support.
This PR also:
* removes CollectDiagnosticsToStringScope from NanobindAdaptors.h. This
was overlooked in a previous PR and it is duplicated in Diagnostics.h.
---------
Co-authored-by: Jacques Pienaar <jpienaar@google.com>
|
|
Reverts revert #118517 after (hopefully) fixing builders
(https://github.com/llvm/llvm-zorg/pull/328,
https://github.com/llvm/llvm-zorg/pull/327)
This reverts commit 61bf308cf2fc32452f14861c102ace89f5f36fec.
|
|
Reverts llvm/llvm-project#117922 because deps aren't met on some of the
post-commit build bots.
|
|
This PR allows out-of-tree dialects to write Python dialect modules
using nanobind instead of pybind11.
It may make sense to migrate in-tree dialects and some of the ODS Python
infrastructure to nanobind, but that is a topic for a future change.
This PR makes the following changes:
* adds nanobind to the CMake and Bazel build systems. We also add
robin_map to the Bazel build, which is a dependency of nanobind.
* adds a PYTHON_BINDING_LIBRARY option to various CMake functions, such
as declare_mlir_python_extension, allowing users to select a Python
binding library.
* creates a fork of mlir/include/mlir/Bindings/Python/PybindAdaptors.h
named NanobindAdaptors.h. This plays the same role, using nanobind
instead of pybind11.
* splits CollectDiagnosticsToStringScope out of PybindAdaptors.h and
into a new header mlir/include/mlir/Bindings/Python/Diagnostics.h, since
it is code that is no way related to pybind11 or for that matter,
Python.
* changed the standalone Python extension example to have both pybind11
and nanobind variants.
* changed mlir/python/mlir/dialects/python_test.py to have both pybind11
and nanobind variants.
Notes:
* A slightly unfortunate thing that I needed to do in the CMake
integration was to use FindPython in addition to FindPython3, since
nanobind's CMake integration expects the Python_ names for variables.
Perhaps there's a better way to do this.
|
|
Make MLIR Python sources idempotent by moving the custom commands off
the custom target for each source.
Previously, a custom target was created for each MLIR Python source file
and a custom command added to symlink it to a destination path. However
this causes the build to not be idempotent because custom targets always
run.
Instead, this PR separates the symlink/copy to destination paths into
custom commands and makes the custom target depend on the output. That
prevents re-running them and restores idempotency.
Testing:
- Build with `-D MLIR_ENABLE_BINDINGS_PYTHON=ON`. Prior to this change,
the build is not idempotent (building twice re-runs the symlink/copy
command for Python source targets). After this change it is idempotent.
|
|
Encountered a problem when linking multiple CAPI libs.
The `COMMON_CAPI_LINK_LIBS` should be multi_value_keywords here.
|
|
This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`.
There are a few parts to this:
1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases.
2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation:
1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet).
2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`).
A few small changes:
1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default).
2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords).
3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case.
Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate.
Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`).
Reviewed By: stellaraccident
Differential Revision: https://reviews.llvm.org/D157934
|
|
Differential Revision: https://reviews.llvm.org/D155362
|
|
This functionality has been replaced by TypeCasters (see D151840)
depends on D154468
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D154469
|
|
The generator expression previously used to enable exceptions would not work since the compiler id of clang-cl is Clang, even if used via clang-cl.
The patch fixes that by replacing the generator expression with simple logic, setting the right compiler flags for all MSVC like compilers (including clang-cl) and all GCC like compilers.
Differential Revision: https://reviews.llvm.org/D141155
|
|
This reverts commit f7a33090a91015836497c75f173775392ab0304d.
Unfortunately this causes a number of failures that didn't show up in my
local build.
|
|
We held off on this before as `LLVM_LIBDIR_SUFFIX` conflicted with it.
Now we return this.
`LLVM_LIBDIR_SUFFIX` is kept as a deprecated way to set
`CMAKE_INSTALL_LIBDIR`. The other `*_LIBDIR_SUFFIX` are just removed
entirely.
I imagine this is too potentially-breaking to make LLVM 15. That's fine.
I have a more minimal version of this in the disto (NixOS) patches for
LLVM 15 (like previous versions). This more expansive version I will
test harder after the release is cut.
Reviewed By: sebastian-ne, ldionne, #libc, #libc_abi
Differential Revision: https://reviews.llvm.org/D130586
|
|
In D128230, we accidentally moved the install for Python sources outside of the loop, having one install() per group of files. While it would be nice if we could do this, it means that we flatten the relative directory tree and every source ends up in the root. The right way to do this is to use FILE_SETS, which preserve the relative directory tree, but they are not available until CMake 3.23.
Differential Revision: https://reviews.llvm.org/D129434
|
|
The refactor in https://reviews.llvm.org/D128230 introduced a new target and the name is not scoped properly, leading to name collisions on larger projects. It is done properly on the target just below, so applying the same pattern here fixes the issue.
|
|
This is already partially the case, but we can rely more heavily on interface libraries and how they are imported/exported in other to simplify the implementation of the mlir python functions in Cmake.
This change also makes a couple of other changes:
1) Add a new CMake function which handles "pure" sources. This was done inline previously
2) Moves the headers associated with CAPI libraries to the libraries themselves. These were previously managed in a separate source target. They can now be added directly to the CAPI libraries using DECLARED_HEADERS.
3) Cleanup some dependencies that showed up as an issue during the refactor
This is a big CMake change that should produce no impact on the build of mlir and on the produced *build tree*. However, this change fixes an issue with the *install tree* of mlir which was previously unusable for projects like torch-mlir because both the "pure" and "extension" targets were pointing to either the build or source trees.
Reviewed By: stellaraccident
Differential Revision: https://reviews.llvm.org/D128230
|
|
Python bindings for extensions of the Transform dialect are defined in separate
Python source files that can be imported on-demand, i.e., that are not imported
with the "main" transform dialect. This requires a minor addition to the
ODS-based bindings generator. This approach is consistent with the current
model for downstream projects that are expected to bundle MLIR Python bindings:
such projects can include their custom extensions into the bundle similarly to
how they include their dialects.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D126208
|
|
Currently, building mlir with the python bindings enabled on Windows in Debug is broken because pybind11, python and cmake don't like to play together. This change normalizes how the three interact, so that the builds can now run and succeed.
The main issue is that python and cmake both make assumptions about which libraries are needed in a Windows build based on the flavor.
- cmake assumes that a debug (or a debug-like) flavor of the build will always require pythonX_d.lib and provides no option/hint to tell it to use a different library. cmake does find both the debug and release versions, but then uses the debug library.
- python (specifically pyconfig.h and by extension python.h) hardcodes the dependency on pythonX_d.lib or pythonX.lib depending on whether `_DEBUG` is defined. This is NOT transparent - it does not show up anywhere in the build logs until the link step fails with `pythonX_d.lib is missing` (or `pythonX.lib is missing`)
- pybind11 tries to "fix" this by implementing a workaround - unless Py_DEBUG is defined, `_DEBUG` is explicitly undefined right before including python headers. This also requires some windows headers to be included differently, so while clever, this is a non-trivial workaround.
mlir itself includes the pybind11 headers (which contain the workaround) AS WELL AS python.h, essentially always requiring both pythonX.lib and pythonX_d.lib for linking. cmake explicitly only adds one or the other, so the build fails.
This change does a couple of things:
- In the cmake files, explicitly add the release version of the python library on Windows builds regardless of flavor. Since Py_DEBUG is not defined, pybind11 will always require release and it will be satisfied
- To satisfy python as well, this change removes any explicit inclusions of Python.h on Windows instead relying on the fact that pybind11 headers will bring in what is needed
There are a few additional things that we could do but I rejected as unnecessary at this time:
- define Py_DEBUG based on the CMAKE_BUILD_TYPE - this will *mostly* work, we'd have to think through multiconfig generators like VS, but it's possible. There doesn't seem to be a need to link against debug python at the moment, so I chose not to overcomplicate the build and always default to release
- similar to above, but define Py_DEBUG based on the CMAKE_BUILD_TYPE *as well as* the presence of the debug python library (`Python3_LIBRARY_DEBUG`). Similar to above, this seems unnecessary right now. I think it's slightly better than above because most people don't actually have the debug version of python installed, so this would prevent breaks in that case.
- similar to the two above, but add a cmake variable to control the logic
- implement the pybind11 workaround directly in mlir (specifically in Interop.h) so that Python.h can still be included directly. This seems prone to error and a pain to maintain in lock step with pybind11
- reorganize how the pybind11 headers are included and place at least one of them in Interop.h directly, so that the header has all of its dependencies included as was the original intention. I decided against this because it really doesn't need pybind11 logic and it's always included after pybind11 is, so we don't necessarily need the python includes
Reviewed By: stellaraccident
Differential Revision: https://reviews.llvm.org/D125284
|
|
There are a couple of issues with the python bindings on Windows:
- `create_symlink` requires special permissions on Windows - using `copy_if_different` instead allows the build to complete and then be usable
- the path to the `python_executable` is likely to contain spaces if python is installed in Program Files. llvm's python substitution adds extra quotes in order to account for this case, but mlir's own python substitution does not
- the location of the shared libraries is different on windows
- if the type is not specified for numpy arrays, they appear to be treated as strings
I've implemented the smallest possible changes for each of these in the patch, but I would actually prefer a slightly more comprehensive fix for the python_executable and the shared libraries.
For the python substitution, I think it makes sense to leverage the existing %python instead of adding %PYTHON and instead add a new variable for the case when preloading is needed. This would also make it clearer which tests are which and should be skipped on platforms where the preloading won't work.
For the shared libraries, I think it would make sense to pass the correct path and extension (possibly even the names) to the python script since these are known by lit and don't have to be hardcoded in the test at all.
Reviewed By: stellaraccident
Differential Revision: https://reviews.llvm.org/D125122
|
|
Re-applies D111513:
* Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled).
* Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing.
* Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API.
* Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan).
* Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time).
* Downstreams will need to adapt by:
* Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory.
* To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension.
This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755.
Reviewed By: stephenneuendorffer
Differential Revision: https://reviews.llvm.org/D113732
|
|
This reverts commit c7be8b75399c727ec9e1ddc3f81510f284c65155.
Build is broken (multiple buildbots)
|
|
* Depends on D111504, which provides the boilerplate for building aggregate shared libraries from installed MLIR.
* Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled).
* Rips out remnants of custom extension building in favor of `pybind11_add_module` which does the right thing.
* Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API.
* Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan).
* Adds a ROOT_DIR argument to `declare_mlir_python_extension` since now all C++ sources for an extension must be under the same directory (to line up at install time).
* Need to validate against a downstream or two and adjust, prior to submitting.
Downstreams will need to adapt by:
* Remove absolute paths from any SOURCES for `declare_mlir_python_extension` (I believe all downstreams are just using `${CMAKE_CURRENT_SOURCE_DIR}` here, which can just be ommitted). May need to set `ROOT_DIR` if not relative to the current source directory.
* To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for `declare_mlir_python_extension`.
Reviewed By: stephenneuendorffer, mikeurbach
Differential Revision: https://reviews.llvm.org/D111513
|
|
installed LLVM.
* Incorporates a reworked version of D106419 (which I have closed but has comments on it).
* Extends the standalone example to include a minimal CAPI (for registering its dialect) and a test which, from out of tree, creates an aggregate dylib and links a little sample program against it. This will likely only work today in *static* MLIR builds (until the TypeID fiasco is finally put to bed). It should work on all platforms, though (including Windows - albeit I haven't tried this exact incarnation there).
* This is the biggest pre-requisite to being able to build out of tree MLIR Python-based projects from an installed MLIR/LLVM.
* I am rather nauseated by the CMake shenanigans I had to endure to get this working. The primary complexity, above and beyond the previous patch is because (with no reason given), it is impossible to export target properties that contain generator expressions... because, of course it isn't. In this case, the primary reason we use generator expressions on the individual embedded libraries is to support arbitrary ordering. Since that need doesn't apply to out of tree (which import everything via FindPackage at the outset), we fall back to a more imperative way of doing the same thing if we detect that the target was imported. Gross, but I don't expect it to need a lot of maintenance.
* There should be a relatively straight-forward path from here to rebase libMLIR.so on top of this facility and also make it include the CAPI.
Differential Revision: https://reviews.llvm.org/D111504
|
|
After CMake 3.18, we are able to limit the scope of the search to just
Development.Module. Searching for Development will fail in situations
where the Python libraries are not available. When possible, limit to
just Development.Module. See:
https://pybind11.readthedocs.io/en/stable/compiling.html#findpython-mode
Reviewed By: stellaraccident
Differential Revision: https://reviews.llvm.org/D111585
|
|
* Now that packaging has stabilized, removes old mechanisms for loading extensions, preferring direct importing.
* Removes _cext_loader.py, _dlloader.py as unnecessary.
* Fixes the path where the CAPI dll is written on Windows. This enables that path of least resistance loading behavior to work with no further drama (see: https://bugs.python.org/issue36085).
* With this patch, `ninja check-mlir` on Windows with Python bindings works for me, modulo some failures that are actually due to a couple of pre-existing Windows bugs. I think this is the first time the Windows Python bindings have worked upstream.
* Downstream changes needed:
* If downstreams are using the now removed `load_extension`, `reexport_cext`, etc, then those should be replaced with normal import statements as done in this patch.
Reviewed By: jdd, aartbik
Differential Revision: https://reviews.llvm.org/D108489
|
|
MSVC needs to know where to put the archive (.lib) as well as the runtime
(.dll). If left to the default location, multiple rules to generate the same
file will be produced, creating a Ninja error.
Differential Revision: https://reviews.llvm.org/D108181
|
|
* Broken out of https://reviews.llvm.org/D106419
* Otherwise, same named modules will conflict with each other.
Differential Revision: https://reviews.llvm.org/D107113
|
|
Differential Revision: https://reviews.llvm.org/D107100
|
|
* Adds source targets (not included in the full set that downstreams use by default) to bundle mlir-c/ headers into the mlir/_mlir_libs/include directory.
* Adds a minimal entry point to get include and library directories.
* Used by npcomp to export a full CAPI (which is then used by the Torch extension to link npcomp).
Reviewed By: mikeurbach
Differential Revision: https://reviews.llvm.org/D107090
|
|
* Implements all of the discussed features:
- Links against common CAPI libraries that are self contained.
- Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API.
- Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API.
- Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done.
- When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al.
- Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues.
- Begins modularizing the API so that external projects that just include a piece pay only for what they use.
* I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes.
* Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions).
* This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing.
* Fixed a couple of absolute imports that were causing issues.
Differential Revision: https://reviews.llvm.org/D106520
|
