CanonicalizePath: Remove kMaxComponents limit #2358
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch refactors the
CanonicalizePath()
function to fix two issues and improve performance. This is achieved through the following:Remove the kMaxComponents limit entirely, which fixes choking on lots of short paths (ninja: fatal: path has too many components) #1732, by dropping the
components
array entirely, in favor of back-tracking the destination pointer.Properly handle
/
and\\
which were incorrectly converted into an empty string. This fixes ninja thinks the root directory doesn't exist #2008.Skip initial
../
components in relative paths, as these are common when referencing source files in build plans, and doing so make the loop after this step run faster in practice since most source files do not need adjustments.Simplify the inner loop logic by handling the last component (which is not followed by a trailing directory separator) separately.
This noticeably improves performance because the inner loop becomes smaller with less branch mis-predictions in the general case.
Never access or copy the character after the end of the input string.
Use
memchr()
to find the next/
on Posix, which allows the use of SIMD implementations provided by the C runtime (e.g. through IFUNC functions on Linux), resulting in very noticeable speedup. This is also why a statically Ninja executable will be slower than one that links to the C library dynamically :-/Avoid performing any writes when the input path doesn't need any adjustment, which is also quite common.
Note that this patch does not remove the 64-bit limit for the
slash_bits
value, which is only used on Win32.Benchmarking was done in several ways:
On Linux, running
hyperfine canon-perftest
to run the canonicalization benchmark program and measure its total running time. Three compilers were used to generate dynamically-linked executables.On Windows, running
canon-perftest
5 times and keeping the best reported average result. The number are slower since they only measure the benched function.On Linux, run
hyperfine ninja -C out/default -n --quiet
on a large Fuchsia build plan, once with 70000+ pending commands, and once after the build (i.e.ninja: no work to do
).