| Age | Commit message (Collapse) | Author |
|---|
|
These are identified by misc-include-cleaner. I've filtered out those
that break builds. Also, I'm staying away from llvm-config.h,
config.h, and Compiler.h, which likely cause platform- or
compiler-specific build failures.
|
|
Prior to this PR, failing to get the host ID would leave the unique lock
file on the file system. This is now fixed by constructing
`RemoveUniqueLockFileOnSignal` earlier. This PR also removes one call to
`sys::fs::remove()` that is now redundant and another that was redundant
even before this patch.
|
|
This PR abstracts the `LockFileManager` API into new `AdvisoryLock`
interface. This is so that we can create an alternative implementation
for Clang implicitly-built modules that is optimized for single-process
environment.
|
|
This patch removes some internal state out of `LockFileManager` by
moving the locking code from the constructor into new member function
`tryLock()` which returns the errors right away. This simplifies and
modernizes the interface.
|
|
Currently, `LockFileManager` assumes the owner of the lock file creates
an output file. This is problematic for at least three reasons:
1. It is orthogonal to the main purpose of this class - mutual
exclusion. This makes creating an alternative implementation more
complicated than it needs to be.
2. Some clients (like the upstream `AMDGPUSplitModule.cpp` file) assume
the output file is not necessary. The owner of the lock file does not
write the file expected by `LockFileManager` and the processes waiting
for the non-owned lock file to be unlocked therefore assume the owner
has died. This means that the work gets repeated by each waiting
process, serially.
3. The documentation makes it sound like successfully waiting for a
non-owned lock file guarantees the output file to be present on the file
system. Implicitly-built modules rely on this. However, the module file
may disappear between `LockFileManager` performing the check and the
compiler loading the module (for example due to module cache pruning
with short intervals, or intervention from outside of Clang). The
compiler assumes this cannot happen, and fails the build if it does.
This PR solves this situation by removing the check, reflecting that in
the `LockFileManager` documentation, and fixing the time-of-check
time-of-use bug in implicit modules.
|
|
Identified with misc-include-cleaner.
|
|
Without these explicit includes, removing other headers, who implicitly
include llvm-config.h, may have non-trivial side effects.
|
|
Since `raw_string_ostream` doesn't own the string buffer, it is
desirable (in terms of memory safety) for users to directly reference
the string buffer rather than use `raw_string_ostream::str()`.
Work towards TODO comment to remove `raw_string_ostream::str()`.
|
|
|
|
LLVM is inconsistent about how it converts `errno` to `std::error_code`.
This can cause problems because values outside of `std::errc` compare
differently if one is system and one is generic on POSIX systems.
This is even more of a problem on Windows where use of the system
category is just wrong, as that is for Windows errors, which have a
completely different mapping than POSIX/generic errors. This patch fixes
one instance of this mistake in `JSONTransport.cpp`.
This patch adds `errnoAsErrorCode()` which makes it so people do not
need to think about this issue in the future. It also cleans up a lot of
usage of `errno` in LLVM and Clang.
|
|
report_fatal_error (#83655)
As per the comment in `raw_fd_ostream`'s destructor, you must call
`clear_error()` to prevent a call to `report_fatal_error()`. There's not
really a way to test this, but we did encounter it in the wild.
rdar://117347895
|
|
This provides a simple way to implement exponential backoff using a do
while loop.
Usage example (also see the change to LockFileManager.cpp):
```
ExponentialBackoff Backoff(10s);
do {
if (tryToDoSomething())
return ItWorked;
} while (Backoff.waitForNextAttempt());
return Timeout;
```
Abstracting this out of `LockFileManager` as the module build daemon
will need it.
|
|
|
|
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
|
|
These source files no longer use None, so they do not need to include
None.h.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
|
|
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
|
|
That macro was being defined but not used anywhere in libc++, so it
must be safe to remove it.
As a fly-by fix, also remove mentions of this macro in other places
in LLVM, to make sure they were not depending on the value defined in
libc++.
Differential Revision: https://reviews.llvm.org/D110289
|
|
Differential Revision: https://reviews.llvm.org/D78022
|
|
Summary:
When building a large Xcode project with multiple module dependencies, and mixed Objective-C & Swift, I observed a large number of clang processes stalling at zero CPU for 30+ seconds throughout the build. This was especially prevalent on my 18-core iMac Pro.
After some sampling, the major cause appears to be the lock file implementation for precompiled modules in the module cache. When the lock is heavily contended by multiple clang processes, the exponential backoff runs in lockstep, with some of the processes sleeping for 30+ seconds in order to acquire the file lock.
In the attached patch, I implemented a more aggressive polling mechanism that limits the sleep interval to a max of 500ms, and randomizes the wait time. I preserved a limited form of exponential backoff. I also updated the code to use cross-platform timing, thread sleep, and random number capabilities available in C++11.
On iMac Pro (2.3 GHz Intel Xeon W, 18 core):
Xcode 11.1 bundled clang:
502.2 seconds (average of 5 runs)
Custom clang build with LockFileManager patch applied:
276.6 seconds (average of 5 runs)
This is a 1.82x speedup for this use case.
On MacBook Pro (4 core 3.1GHz Intel i7):
Xcode 11.1 bundled clang:
539.4 seconds (average of 2 runs)
Custom clang build with LockFileManager patch applied:
509.5 seconds (average of 2 runs)
As expected, machines with fewer cores benefit less from this change.
```
Call graph:
2992 Thread_393602 DispatchQueue_1: com.apple.main-thread (serial)
2992 start (in libdyld.dylib) + 1 [0x7fff6a1683d5]
2992 main (in clang) + 297 [0x1097a1059]
2992 driver_main(int, char const**) (in clang) + 2803 [0x1097a5513]
2992 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (in clang) + 1608 [0x1097a7cc8]
2992 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (in clang) + 3299 [0x1097dace3]
2992 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (in clang) + 509 [0x1097dcc1d]
2992 clang::FrontendAction::Execute() (in clang) + 42 [0x109818b3a]
2992 clang::ParseAST(clang::Sema&, bool, bool) (in clang) + 185 [0x10981b369]
2992 clang::Parser::ParseFirstTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&) (in clang) + 37 [0x10983e9b5]
2992 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&) (in clang) + 141 [0x10983ecfd]
2992 clang::Parser::ParseExternalDeclaration(clang::Parser::ParsedAttributesWithRange&, clang::ParsingDeclSpec*) (in clang) + 695 [0x10983f3b7]
2992 clang::Parser::ParseObjCAtDirectives(clang::Parser::ParsedAttributesWithRange&) (in clang) + 637 [0x10a9be9bd]
2992 clang::Parser::ParseModuleImport(clang::SourceLocation) (in clang) + 170 [0x10c4841ba]
2992 clang::Parser::ParseModuleName(clang::SourceLocation, llvm::SmallVectorImpl<std::__1::pair<clang::IdentifierInfo*, clang::SourceLocation> >&, bool) (in clang) + 503 [0x10c485267]
2992 clang::Preprocessor::Lex(clang::Token&) (in clang) + 316 [0x1098285cc]
2992 clang::Preprocessor::LexAfterModuleImport(clang::Token&) (in clang) + 690 [0x10cc7af62]
2992 clang::CompilerInstance::loadModule(clang::SourceLocation, llvm::ArrayRef<std::__1::pair<clang::IdentifierInfo*, clang::SourceLocation> >, clang::Module::NameVisibilityKind, bool) (in clang) + 7989 [0x10bba6535]
2992 compileAndLoadModule(clang::CompilerInstance&, clang::SourceLocation, clang::SourceLocation, clang::Module*, llvm::StringRef) (in clang) + 296 [0x10bba8318]
2992 llvm::LockFileManager::waitForUnlock() (in clang) + 91 [0x10b6953ab]
2992 nanosleep (in libsystem_c.dylib) + 199 [0x7fff6a22c914]
2992 __semwait_signal (in libsystem_kernel.dylib) + 10 [0x7fff6a2a0f32]
```
Differential Revision: https://reviews.llvm.org/D69575
|
|
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
|
|
Patch by Xi Ge!
|
|
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
|
|
For the _WIN32 macro, it is the definedness that matters rather than
the value. Most uses of the macro already rely on the definedness.
This commit fixes the few remaining uses that relied on the value.
Differential Revision: https://reviews.llvm.org/D51105
llvm-svn: 340520
|
|
On Windows we've observed that if you open a file, write to it, map it into
memory and close the file handle, the contents of the memory mapping can
sometimes be incorrect. That was what we did when adding an entry to the
ThinLTO cache using the TempFile and MemoryBuffer classes, and it was causing
intermittent build failures on Chromium's ThinLTO bots on Windows. More
details are in the associated Chromium bug (crbug.com/786127).
We can prevent this from happening by keeping a handle to the file open while
the mapping is active. So this patch changes the mapped_file_region class to
duplicate the file handle when mapping the file and close it upon unmapping it.
One gotcha is that the file handle that we keep open must not have been
created with FILE_FLAG_DELETE_ON_CLOSE, as otherwise the operating system
will prevent other processes from opening the file. We can achieve this
by avoiding the use of FILE_FLAG_DELETE_ON_CLOSE altogether. Instead,
we use SetFileInformationByHandle with FileDispositionInfo to manage the
delete-on-close bit. This lets us remove the hack that we used to use to
clear the delete-on-close bit on a file opened with FILE_FLAG_DELETE_ON_CLOSE.
A downside of using SetFileInformationByHandle/FileDispositionInfo as
opposed to FILE_FLAG_DELETE_ON_CLOSE is that it prevents us from using
CreateFile to open the file while the flag is set, even within the same
process. This doesn't seem to matter for almost every client of TempFile,
except for LockFileManager, which calls sys::fs::create_link to create a
hard link from the lock file, and in the process of doing so tries to open
the file. To prevent this change from breaking LockFileManager I changed it
to stop using TempFile by effectively reverting r318550.
Differential Revision: https://reviews.llvm.org/D48051
llvm-svn: 334630
|
|
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.
Patch produced by
for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done
Differential Revision: https://reviews.llvm.org/D46290
llvm-svn: 331272
|
|
See r331124 for how I made a list of files missing the include.
I then ran this Python script:
for f in open('filelist.txt'):
f = f.strip()
fl = open(f).readlines()
found = False
for i in xrange(len(fl)):
p = '#include "llvm/'
if not fl[i].startswith(p):
continue
if fl[i][len(p):] > 'Config':
fl.insert(i, '#include "llvm/Config/llvm-config.h"\n')
found = True
break
if not found:
print 'not found', f
else:
open(f, 'w').write(''.join(fl))
and then looked through everything with `svn diff | diffstat -l | xargs -n 1000 gvim -p`
and tried to fix include ordering and whatnot.
No intended behavior change.
llvm-svn: 331184
|
|
LLVM_ON_WIN32 is set exactly with MSVC and MinGW (but not Cygwin) in
HandleLLVMOptions.cmake, which is where _WIN32 defined too. Just use the
default macro instead of a reinvented one.
See thread "Replacing LLVM_ON_WIN32 with just _WIN32" on llvm-dev and cfe-dev.
No intended behavior change.
This moves over all uses of the macro, but doesn't remove the definition
of it in (llvm-)config.h yet.
llvm-svn: 331127
|
|
No functionality change intended.
llvm-svn: 325460
|
|
This move some of the complexity over to the lower level TempFile.
It also makes it a bit more explicit where errors are ignored since we
now have a call to consumeError.
llvm-svn: 318550
|
|
std::error_code can represent success, so we don't need a
Optional<std::error_code>.
Rename the variable to avoid confusion with the type Error.
llvm-svn: 318111
|
|
llvm-svn: 318104
|
|
Summary:
Previously, we would emit error messages like "IO failure on output
stream". This change causes use to include information about what
actually went wrong, e.g. "No space left on device".
Reviewers: sunfish, rnk
Reviewed By: rnk
Subscribers: mehdi_amini, llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D39203
llvm-svn: 316404
|
|
I did this a long time ago with a janky python script, but now
clang-format has built-in support for this. I fed clang-format every
line with a #include and let it re-sort things according to the precise
LLVM rules for include ordering baked into clang-format these days.
I've reverted a number of files where the results of sorting includes
isn't healthy. Either places where we have legacy code relying on
particular include ordering (where possible, I'll fix these separately)
or where we have particular formatting around #include lines that
I didn't want to disturb in this patch.
This patch is *entirely* mechanical. If you get merge conflicts or
anything, just ignore the changes in this patch and run clang-format
over your #include lines in the files.
Sorry for any noise here, but it is important to keep these things
stable. I was seeing an increasing number of patches with irrelevant
re-ordering of #include lines because clang-format was used. This patch
at least isolates that churn, makes it easy to skip when resolving
conflicts, and gets us to a clean baseline (again).
llvm-svn: 304787
|
|
Go back to behavior pre-r231309 and reduce the timeout from 8 to ~1.5
min now that we have (a) PCMCache mechanism (r298165) and (b) timeout
that doesn't cause a failure, but actually build the module (r298175).
rdar://problem/30297862
llvm-svn: 298176
|
|
llvm-svn: 281450
|
|
other minor fixes.
Differential revision: https://reviews.llvm.org/D23789
llvm-svn: 279535
|
|
This is currently used by clang to lock access to modules; improve the
error message so that clang can use better output messages from locking
error issues.
rdar://problem/26529101
Differential Review: http://reviews.llvm.org/D20942
llvm-svn: 271755
|
|
Reapplies r241005 after fixing the build on non-Mac platforms. Original
commit message below.
The hostname can be very unstable when there are many machines on the
network competing for the same name. Using the hardware UUID makes it
less likely to have collisions or to consider files written by the
current host to be owned by a different one at a later time.
rdar://problem/21512307
llvm-svn: 241012
|
|
Broke non-Mac builds.
This reverts commit r241005.
llvm-svn: 241007
|
|
The hostname can be very unstable when there are many machines on the
network competing for the same name. Using the hardware UUID makes it
less likely to have collisions or to consider files written by the
current host to be owned by a different one at a later time.
rdar://problem/21512307
llvm-svn: 241005
|
|
Make sure to remove the unique lock file, which is what the .lock
symlink points to, if there is a signal while the lock is held. This
will release the lock, since the symlink will point to nothing (already
tested in unit tests). For good measure, also clean up the unique lock
file if there is an error or signal before the lock is acquired.
I will add a clang test.
rdar://problem/21512307
llvm-svn: 240967
|
|
NFC.
llvm-svn: 232976
|
|
Now that SmallString is a first-class citizen, most SmallString::str()
calls are not required. This patch removes a whole bunch of them, yet
there are lots more.
There are two use cases where str() is really needed:
1) To use one of StringRef member functions which is not available in
SmallString.
2) To convert to std::string, as StringRef implicitly converts while
SmallString do not. We may wish to change this, but it may introduce
ambiguity.
llvm-svn: 232622
|
|
Waiting for just 1 min may not be enough for some contexts.
llvm-svn: 231309
|
|
This is true in clang, and let's us remove the problematic code that
waits around for the original file and then times out if it doesn't get
created in short order. This caused any 'dead' lock file or legitimate
time out to cause a cascade of timeouts in any processes waiting on the
same lock (even if they only just showed up).
llvm-svn: 229881
|
|
5 minutes is an eternity, so try to strike a better balance between
waiting long enough for any reasonable module build and not so long that
users kill the process because they think it's hanging.
Also give the client a way to delete the lock file after a timeout.
llvm-svn: 228603
|
|
The main difference is the removal of
std::error_code exists(const Twine &path, bool &result);
It was an horribly redundant interface since a file not existing is also a valid
error_code. Now we have an access function that returns just an error_code. This
is the only function that has to be implemented for Unix and Windows. The
functions can_write, exists and can_execute an now just wrappers.
One still has to be very careful using these function to avoid introducing
race conditions (Time of check to time of use).
llvm-svn: 217625
|
|
Instead of moving out the data in a ErrorOr<std::unique_ptr<Foo>>, get
a reference to it.
Thanks to David Blaikie for the suggestion.
llvm-svn: 214516
|
|
llvm-svn: 212405
|
|
While std::error_code itself seems to work OK in all platforms, there
are few annoying differences with regards to the std::errc enumeration.
This patch adds a simple llvm enumeration, which will hopefully avoid build
breakages in other platforms and surprises as we get more uses of
std::error_code.
llvm-svn: 210920
|
