Add a prologue to the kernel entry to handle cases where code designed for kernarg preloading is executed on hardware equipped with incompatible firmware. If hardware has compatible firmware the 256 bytes at the start of the kernel entry will be skipped. This skipping is done automatically by hardware that supports the feature. A pass is added which is intended to be run at the very end of the pipeline to avoid any optimizations that would assume the prologue is a real predecessor block to the actual code start. In reality we have two possible entry points for the function. 1. The optimized path that supports kernarg preloading which begins at an offset of 256 bytes. 2. The backwards compatible entry point which starts at offset 0.
The assumption in the asserts is based on the fact that no SGPR/VGPR register Arg mask in the ISelLowering and Legalizer can equal zero. They are implicitly set to ~0 by default (meaning non-masked) or explicitly to a non-zero value. The `optimizeCompareInstr` case is different from the above described. It requires the mask to be a power-of-two because it's a special-case optimization, hence in this case we still cannot have an invalid shift. This commit also silences static analysis tools wrt potential bad shifts that could result from the output of `countr_zero(Mask)`.
The actual size of scratch/private is determined at dispatch time, so add more plumbing to request it. Will be used in subsequent change.
Reformat one table and protect a couple of tables that we don't want to reformat.
This patch adds the DAG isel changes for kernel argument preloading. These changes are not usable with older firmware but subsequent patches in the series will make the codegen backwards compatible. This patch should only be submitted alongside that subsequent patch. Preloading here begins from the start of the kernel arguments until the amount of arguments indicated by the CL flag amdgpu-kernarg-preload-count. Aggregates and arguments passed by-ref are not supported. Special care for the alignment of the kernarg segment is needed as well as consideration of the alignment of addressable SGPR tuples when we cannot directly use misaligned large tuples that the arguments are loaded to. Reviewed By: bcahoon Differential Revision: https://reviews.llvm.org/D158579
Implement an intrinsic for use lowering LDS variables to different addresses from different kernels. This will allow kernels that cannot reach an LDS variable to avoid wasting space for it. There are a number of implicit arguments accessed by intrinsic already so this implementation closely follows the existing handling. It is slightly novel in that this SGPR is written by the kernel prologue. It is necessary in the general case to put variables at different addresses such that they can be compactly allocated and thus necessary for an indirect function call to have some means of determining where a given variable was allocated. Claiming an arbitrary SGPR into which an integer can be written by the kernel, in this implementation based on metadata associated with that kernel, which is then passed on to indirect call sites is sufficient to determine the variable address. The intent is to emit a __const array of LDS addresses and index into it. Reviewed By: arsenm Differential Revision: https://reviews.llvm.org/D125060
Differential Revision: https://reviews.llvm.org/D96906
Reviewed By: rampitec Differential Revision: https://reviews.llvm.org/D93813
Attempt to fix build error with ancient GCC
When passing special ABI inputs, we have no existing context for the type to use.