Consider the remaining job capacity in main loop

[Flowdalic]

This changes CanRunMore() to return an int instead of a bool. The
return value is the remaining capacity of new jobs that can be spawned
without saturating a potential enabled load limitation (if ninja's -l
option is used). We assume that every started edge increases the load
by one, hence the available "load capacity" is the maximum load minus
the current load.

So, instead of spawning new jobs only limited by the remaining job
capacity ("-j") if the load average limitation is not yet reached, we
only spawn so many new jobs that would saturate the load limitation.

This helps to speed up multiple parallel builds on the same host, as
the following benchmark shows. We compare the total build times of 8
parallel builds of LLVM on a 256-core system using "ninja -j 258".

ninja-master: 1351 seconds
ninja-load-capacity: 920 seconds

That is, with this commit, the whole process becomes 1.46× faster.

The used benchmark script created and prepared 8 build directories,
records the start time, spawns 8 subshells invoking "ninja -j 258",
awaits the termination of those subshells, and records the end
time. Besides the total running time, it also outputs /proc/loadavg,
provides an indication of where the performance is gained:

ninja-master: 3.90 93.94 146.38 1/1936 209125
ninja-load-capacity: 92.46 210.50 199.90 1/1936 36917

So with this change, ninja uses the available hardware cores better in
the presence of competing ninja processes, while avoiding to overload
the system.

The used benchmark script for reference:

#!/usr/bin/env bash
set -euo pipefail

VANILLA_NINJA=~/code/ninja-master/build/ninja
LOAD_CAPACITY_AWARE_NINJA=~/code/ninja-load-capacity/build/ninja

CMAKE_NINJA_PROJECT_SOURCE=~/code/llvm-project/llvm

declare -ir PARALLEL_BUILDS=8
readonly TMP_DIR=$(mktemp --directory --tmpdir=/var/tmp)

cleanup() {
	rm -rf "${TMP_DIR}"
}
trap cleanup EXIT

BUILD_DIRS=()
for i in $(seq 1 ${PARALLEL_BUILDS}); do
	BUILD_DIR="${TMP_DIR}/${i}"
	mkdir "${BUILD_DIR}"
	(
		cd "${BUILD_DIR}"
		cmake -G Ninja "${CMAKE_NINJA_PROJECT_SOURCE}" &> "${BUILD_DIR}/build.log"
	)&
	BUILD_DIRS+=("${BUILD_DIR}")
done
wait

NPROC=$(nproc)
MAX_LOAD=$(( NPROC + 2 ))
SLEEP_SECONDS=300

for NINJA_BIN in "${VANILLA_NINJA}" "${LOAD_CAPACITY_AWARE_NINJA}"; do
	for BUILD_DIR in "${BUILD_DIRS[@]}"; do
		(
			"${NINJA_BIN}" -C "${BUILD_DIR}" clean &> "${BUILD_DIR}/build.log"
		)&
	done
	wait

	echo "Starting build with ${NINJA_BIN} using -j ${MAX_LOAD}"
	START=$(date +%s)
	for BUILD_DIR in "${BUILD_DIRS[@]}"; do
		(
			"${NINJA_BIN}" -C "${BUILD_DIR}" -l "${MAX_LOAD}" &> "${BUILD_DIR}/build.log"
		)&
	done
	wait
	STOP=$(date +%s)

	DELTA_SECONDS=$((STOP - START))
	echo "Using ${NINJA_BIN} to perform ${PARALLEL_BUILDS} of ${CMAKE_NINJA_PROJECT_SOURCE}"
	echo "took ${DELTA_SECONDS} seconds on this ${NPROC} core system using -j ${MAX_LOAD}"
	echo "/proc/loadavg:"
	cat /proc/loadavg
	echo "ninja --version:"
	"${NINJA_BIN}" --version

	echo "Sleeping "${SLEEP_SECONDS}" seconds to bring system into quiescent state"
	sleep ${SLEEP_SECONDS}
done

[Flowdalic]

I am not sure if the Windows build failure is caused by my changes or something else. Since I am not a windows person, any help is appreciated. Thanks. :)

[jhasse]

Why is this faster?

[Flowdalic]

Why is this faster?

Due the reduction of the total number of involved context switches.

I tried to explain the situation better in the new commit message, over realizing that it is maybe not so obvious and that I failed to do so in the initial commit message. Please have a look at the new commit message and let me know if you have any further questions. :)

[jhasse]

Thanks! The new commit message is great.

I'm not a user of the load limit parameter, maybe someone else can comment / review.

The Windows build failure is a general problem (see master branch) which I don't understand either :(

[Flowdalic]

Friendly reminder :)

[Flowdalic]

@jhasse friendly reminder. Anything I can do to get this merged?

[Flowdalic]

Looks like someone needs to approve the CI workflows. Thanks.

[Flowdalic]

@Flowdalic it's not that the CI workflow needs approval.

I am not sure about. I see here

3 workflows awaiting approval
First-time contributors need a maintainer to approve running workflows. Learn more.
1 successful check

/usr/include/c++/4.8.2/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
 #error This file requires compiler and library support for the \

fixed by replacing cstdint with stdint.h.

[Flowdalic]

Squashed everything into a single commit. Someone may want to approve the workflows. Anything else to get this merged?

[Flowdalic]

Friendly Ping. Anything I can do to get this merged?

[Flowdalic]

Another friendly Ping. Anything I can do to get this merged?

[Flowdalic]

Note that I had to revert the suggested changes because ninja does not support C++11 yet.

[Flowdalic]

Friendly ping.

[Flowdalic]

@jhasse thanks for your review. I have addressed your comments.