There's a pattern throughout LLVM of cl::opts being exported. That in itself is probably a bit unfortunate, but what's especially bad about it is that a lot of those symbols are in the global namespace. Move them into the llvm namespace. While doing this, I noticed some other variables in the global namespace and moved them as well.
This is an attempt to reland #151660 by including a missing STL header found by a buildbot failure. The stable function map could be huge for a large application. Fully loading it is slow and consumes a significant amount of memory, which is unnecessary and drastically slows down compilation especially for non-LTO and distributed-ThinLTO setups. This patch introduces an opt-in lazy loading support for the stable function map. The detailed changes are: - `StableFunctionMap` - The map now stores entries in an `EntryStorage` struct, which includes offsets for serialized entries and a `std::once_flag` for thread-safe lazy loading. - The underlying map type is changed from `DenseMap` to `std::unordered_map` for compatibility with `std::once_flag`. - `contains()`, `size()` and `at()` are implemented to only load requested entries on demand. - Lazy Loading Mechanism - When reading indexed codegen data, if the newly-introduced `-indexed-codegen-data-lazy-loading` flag is set, the stable function map is not fully deserialized up front. The binary format for the stable function map now includes offsets and sizes to support lazy loading. - The safety of lazy loading is guarded by the once flag per function hash. This guarantees that even in a multi-threaded environment, the deserialization for a given function hash will happen exactly once. The first thread to request it performs the load, and subsequent threads will wait for it to complete before using the data. For single-threaded builds, the overhead is negligible (a single check on the once flag). For multi-threaded scenarios, users can omit the flag to retain the previous eager-loading behavior.
This reverts commit 76dd742f7b32e4d3acf50fab1dbbd897f215837e.
The stable function map could be huge for a large application. Fully loading it is slow and consumes a significant amount of memory, which is unnecessary and drastically slows down compilation especially for non-LTO and distributed-ThinLTO setups. This patch introduces an opt-in lazy loading support for the stable function map. The detailed changes are: - `StableFunctionMap` - The map now stores entries in an `EntryStorage` struct, which includes offsets for serialized entries and a `std::once_flag` for thread-safe lazy loading. - The underlying map type is changed from `DenseMap` to `std::unordered_map` for compatibility with `std::once_flag`. - `contains()`, `size()` and `at()` are implemented to only load requested entries on demand. - Lazy Loading Mechanism - When reading indexed codegen data, if the newly-introduced `-indexed-codegen-data-lazy-loading` flag is set, the stable function map is not fully deserialized up front. The binary format for the stable function map now includes offsets and sizes to support lazy loading. - The safety of lazy loading is guarded by the once flag per function hash. This guarantees that even in a multi-threaded environment, the deserialization for a given function hash will happen exactly once. The first thread to request it performs the load, and subsequent threads will wait for it to complete before using the data. For single-threaded builds, the overhead is negligible (a single check on the once flag). For multi-threaded scenarios, users can omit the flag to retain the previous eager-loading behavior.
Names are used for debugging purpose and have no impact on codegen. For a non-trivial project, reading them consumes a lot of memory and slows down the compilation significantly. This patch adds a field in the indexed CGData to remember the total size of Names, and creates a command-line option to skip reading Names by advancing the pointer when deserializing the indexed CGData.
These are identified by misc-include-cleaner. I've filtered out those that break builds. Also, I'm staying away from llvm-config.h, config.h, and Compiler.h, which likely cause platform- or compiler-specific build failures.
…Stream. CachedFileStream has previously performed the commit step in its destructor, but this means its only recourse for error handling is report_fatal_error. Modify this to add an explicit commit() method, and call this in the appropriate places with appropriate error handling for the location. Currently the destructor of CacheStream gives an assert failure in Debug builds if commit() was not called. This will help track down any remaining uses of the API that assume the old destructior behaviour. In Release builds we fall back to the previous behaviour and call report_fatal_error if the commit fails. This is version 2 of this PR, superseding reverted PR https://github.com/llvm/llvm-project/pull/115331 . I have incorporated a change to the testcase to make it more reliable on Windows, as well as two follow-up changes (https://github.com/llvm/llvm-project/commit/df79000896101acc9b8d7435e59f767b36c00ac8 and https://github.com/llvm/llvm-project/commit/b0baa1d8bd68a2ce2f7c5f2b62333e410e9122a1) that were also reverted when 115331 was reverted. --------- Co-authored-by: Augie Fackler <augie@google.com> Co-authored-by: Vitaly Buka <vitalybuka@google.com>
This reverts commit ce9e1d3c15ed6290f1cb07b482939976fa8115cd. The unittest added in this commit seems to be flaky causing random failure on buildbots: - https://lab.llvm.org/buildbot/#/builders/46/builds/13235 - https://lab.llvm.org/buildbot/#/builders/46/builds/13232 - https://lab.llvm.org/buildbot/#/builders/46/builds/13228 - https://lab.llvm.org/buildbot/#/builders/46/builds/13224 - https://lab.llvm.org/buildbot/#/builders/46/builds/13220 - https://lab.llvm.org/buildbot/#/builders/46/builds/13210 - https://lab.llvm.org/buildbot/#/builders/46/builds/13208 - https://lab.llvm.org/buildbot/#/builders/46/builds/13207 - https://lab.llvm.org/buildbot/#/builders/46/builds/13202 - https://lab.llvm.org/buildbot/#/builders/46/builds/13196 and - https://lab.llvm.org/buildbot/#/builders/180/builds/14266 - https://lab.llvm.org/buildbot/#/builders/180/builds/14254 - https://lab.llvm.org/buildbot/#/builders/180/builds/14250 - https://lab.llvm.org/buildbot/#/builders/180/builds/14245 - https://lab.llvm.org/buildbot/#/builders/180/builds/14244 - https://lab.llvm.org/buildbot/#/builders/180/builds/14226
…Stream. CachedFileStream has previously performed the commit step in its destructor, but this means its only recourse for error handling is report_fatal_error. Modify this to add an explicit commit() method, and call this in the appropriate places with appropriate error handling for the location. Currently the destructor of CacheStream gives an assert failure in Debug builds if commit() was not called. This will help track down any remaining uses of the API that assume the old destructior behaviour. In Release builds we fall back to the previous behaviour and call report_fatal_error if the commit fails.
#125983