Propagate supbrocess' exit codes to the ninja exit code #2540

Felixoid · 2024-12-03T14:56:04Z

Now, ninjas could exit with codes 0, 1, and 2.

With these codes, it's nearly impossible to get what went wrong in the subcommands.

This patch propagates the subcommand's exit code to the ninja's exit code.

It fixes #1123, fixes #1507, and technically substitutes the #1805

Felixoid · 2024-12-03T15:59:32Z

The reason why do we need it:

In ClickHouse, we use prlimit to identify when some binary becomes too big, but it fails in the entirely same way as a standard failing build.

And here how the prlimit is able to fail in different cases:

$ prlimit --data=20000000 --cpu=2 clickhouse-local -q 'SELECT * FROM (SELECT * FROM system.numbers LIMIT 100000000) GROUP BY number' ; echo $?
Code: 173. DB::ErrnoException: Allocator: Cannot malloc 1.00 MiB.: , errno: 12, strerror: Cannot allocate memory. (CANNOT_ALLOCATE_MEMORY)
173
$ prlimit --data=15000000 --cpu=2 clickhouse-local -q 'SELECT * FROM (SELECT * FROM system.numbers LIMIT 100000000) GROUP BY number' ; echo $?
Segmentation fault (core dumped)
139
$ prlimit --cpu=2 clickhouse-local -q 'SELECT * FROM (SELECT * FROM system.numbers LIMIT 100000000) GROUP BY number' ; echo $?
Killed
137

Felixoid · 2024-12-03T18:14:20Z

After the last push, it works like this:

> ./ninja test-exit; echo $?
[1/1] exit 155
FAILED: test-exit
exit 155
ninja: build stopped: subcommand failed.
155
> ./ninja test-exit; echo $?
[1/1] exit 1
FAILED: test-exit
exit 1
ninja: build stopped: subcommand failed.
1
> ./ninja test-exit; echo $?
[0/1] exit 2
ninja: build stopped: interrupted by user.
130

Felixoid · 2024-12-03T20:48:39Z

The latest commit adds the exit code to the FAILED log line:

> ./ninja prlimit-kill prlimit-allocate prlimit-sigv ; echo $?
[1/3] prlimit --data=1 clickhouse-local -q 'SELECT * FROM (SELECT * FROM system.numbers GROUP BY number LIMIT 100000000000) GROUP BY number'
FAILED[code 139] prlimit-sigv
prlimit --data=1 clickhouse-local -q 'SELECT * FROM (SELECT * FROM system.numbers GROUP BY number LIMIT 100000000000) GROUP BY number'
[2/3] prlimit --data=9999999 clickhouse-local -q 'SELECT * FROM (SELECT * FROM system.numbers GROUP BY number LIMIT 100000000000) GROUP BY number'
FAILED[code 139] prlimit-allocate
prlimit --data=9999999 clickhouse-local -q 'SELECT * FROM (SELECT * FROM system.numbers GROUP BY number LIMIT 100000000000) GROUP BY number'
[3/3] prlimit --cpu=1 clickhouse-local -q 'SELECT * FROM (SELECT * FROM system.numbers GROUP BY number LIMIT 100000000000) GROUP BY number'
FAILED[code 137] prlimit-kill
prlimit --cpu=1 clickhouse-local -q 'SELECT * FROM (SELECT * FROM system.numbers GROUP BY number LIMIT 100000000000) GROUP BY number'
ninja: build stopped: subcommand failed.
137

Felixoid · 2024-12-04T10:20:36Z

Dear @jhasse and @digit-google, could you please leave a comment on whether it makes sense or if there is something left to improve before the merge?

digit-google · 2024-12-05T08:54:03Z

Thank you for your contribution. I am very busy this week, but I'll take a look the next one. Sorry about that.

digit-google · 2024-12-09T17:23:56Z

Thank you very much for you PR. This feature will be useful.

I think that keeping a dedicated type for the exit status will help keeping the code easier to read and understand. Besides, native exit codes on Win32 are unsigned long, even though the exit() function invoked from Ninja still takes an int. I suggest to achieve this by:

Using C++11 enum sizing with platform-specific types, as in:

// from exit_status.h

// The underlying type of the ExitStatus enum, used to represent a platform-specific
// process exit code.
#ifdef _WIN32
#define EXIT_STATUS_TYPE unsigned long
#else  // !_WIN32
#define EXIT_STATUS_TYPE int
#endif  // !_WIN32

enum ExitStatus : EXIT_STATUS_TYPE {
  ExitSuccess = 0,
  ExitFailure,
  ExitInterrupted,
};

Modify Builder::Build() to return an ExitStatus value instead of a boolean.

Also, on Posix, ensure that an interrupted build returns static_cast<ExitStatus>(130) directly, to get rid of the hack on line 1580 of ninja.cc to do that.

Only cast the ExitStatus to an int when needed, i.e. when calling exit() from ninja.cc. This means NinjaMain::RunBuild() would also return an ExitStatus value instead of an int.

Apart from that, can I ask you to rebase your commits to remove temporary fixes? Nobody really needs this information but you, and it makes reviews, and later looking at the git history, difficult. Ideally tests should be introduced with the modifications they are related to. In this specific case, I would recommend one commit to propagate the status (+tests), then another commit to change the printed status line in case of failure to include the code.

Finally add a regression test in output_test.py for each commit's changes. I.e. one that verifies the value returned by Ninja when invoking a command that fails voluntarily, then another one that checks the text printed by Ninja in case of failure to verify it includes the right code.

There is a tiny chance that the status line change is going to break existing parsing scripts. I recommend you try to keep the original FAILED: sub-string in the output in case of failure to minimize that. I.e. print FAILED: [code=<value>] instead of FAILED[code=<value>]:. @jhasse what do you think about this specific problem?

Felixoid · 2024-12-09T18:06:03Z

Hi @digit-google, thanks for the review!

I think the only unclear thing to me is how to use ExitStatus for other codes than are defined there. I think, you mean using EXIT_STATUS_TYPE in all the places? Than, looks good to me.

I'll apply all the fixes

digit-google · 2024-12-09T18:18:32Z

Can you clarify what you mean. It should be straightforward to just use the ExitStatus type for return values and local/member variables types. Use a static cast when you need to convert an int or DWORD into an ExitStatus in the subprocess Finish() functions, and that should be enough. Wdyt?

Felixoid · 2024-12-09T18:26:14Z

Just in case I am not a real C++ dev 👉👈, I could miss the point.
The idea is to exit Ninja with one of the exit codes of the subprocesses. So, return static_cast<ExitStatus>(42); should still work, right?

digit-google · 2024-12-09T18:41:30Z

It's the other way around, lines 1804-1807 of ninja.cc look like this:

    int result = ninja.RunBuild(argc, argv, status);
    if (g_metrics)
      ninja.DumpMetrics();
    exit(result);

What I am suggesting is having NinjaMain::RunBuild() return an ExitStatus value, then do:

    ExitStatus status = ninja.RunBuild(argc, argv, status);
    if (g_metrics)
      ninja.DumpMetrics();
    exit(static_cast<int>(status));

Felixoid · 2024-12-09T18:48:46Z

but Finish() should as well return the same code it received from the subprocess, so it will be passed from there to ninja.RunBuild?

-int Subprocess::Finish() {
+ExitStatus Subprocess::Finish() {
   assert(pid_ != -1);
   int status;
   if (waitpid(pid_, &status, 0) < 0)
@@ -166,7 +168,7 @@ int Subprocess::Finish() {

   if (WIFEXITED(status)) {
     // propagate the status transparently
-    return WEXITSTATUS(status);
+    return static_cast<ExitStatus>(WEXITSTATUS(status));
   }
   if (WIFSIGNALED(status)) {
     // Overwrite interrupts to exit code 2
@@ -175,7 +177,7 @@ int Subprocess::Finish() {
       return ExitInterrupted;
   }
   // At this point, we exit with any other signal+128
-  return status | 0x80;
+  return static_cast<ExitStatus>(status | 0x80);
 }

Felixoid · 2024-12-09T23:44:39Z

I think all points are addressed:

ExitStatus is used everywhere to pass the exit codes
output_test.py has tests for both logging and exiting

Besides, I updated all the tests depending on Builder::Build, so they are also aware of the ExitStatus return.

src/build.h

src/exit_status.h

misc/output_test.py

digit-google · 2024-12-10T12:15:21Z

Thanks very much for the latest commits. I added a few review comments, please address them, but this is looking very good so far. Now it's up to @jhasse to add his feedback and approve the PR.

misc/output_test.py

digit-google · 2024-12-10T16:04:51Z

Thank you very much, latest version seems perfect to me.

jhasse

Thanks for the PR!

From my experience you're entering a world of pain as soon as you want to do anything with exit codes outside of the range 0 to 254. Therefore I'm not sure if Ninja should support it, even on Windows.

I will think about it.

jhasse · 2024-12-10T17:44:04Z

misc/output_test.py

 import unittest
 from textwrap import dedent
-import typing as T


Why this change? Probably some Python formatter?

It was done by https://github.com/PyCQA/isort/, yes.

Is it ok to keep it sorted? The same question regarding the other pep8-related changes

jhasse · 2024-12-10T17:44:15Z

misc/output_test.py


 default_env = dict(os.environ)
 default_env.pop('NINJA_STATUS', None)
 default_env.pop('CLICOLOR_FORCE', None)
 default_env['TERM'] = ''
 NINJA_PATH = os.path.abspath('./ninja')

+


this one too

This is the pep-8 requirement "two empty lines before and after function/class definitions"

jhasse · 2024-12-10T17:44:19Z

misc/output_test.py

@@ -120,7 +121,7 @@ def run(
                                                 cwd=self.d.name, env=env)
        except subprocess.CalledProcessError as err:
            if print_err_output:
-              sys.stdout.buffer.write(err.output)
+                sys.stdout.buffer.write(err.output)


this one too

Having mixed alignments in the same file confuses very much. Should I revert it?

misc/output_test.py

src/subprocess-posix.cc

src/subprocess_test.cc

Felixoid · 2024-12-10T20:34:57Z

Addressed most of the questions and made review-related changes as fixups to track the progress.

Felixoid · 2024-12-13T10:57:30Z

Can I kindly get another review round? If the things are addressed, I'd rebase everything into proper commits.

Make all the methods return ExitStatus: Subprocess::Finish-> CommandRunner::Result.status-> Builder::Build-> NinjaMain::RunBuild Ninja now return different exit codes for Win32 and Posix systems: - Win32 -> 2 - Posix -> 130 (128+2)

Felixoid · 2024-12-16T11:36:47Z

I've rebased all the changes, let me know if something here is still blocker to proceed 🙏

Felixoid force-pushed the master branch 2 times, most recently from 64c6798 to 91bffbe Compare December 3, 2024 15:09

Felixoid mentioned this pull request Dec 3, 2024

Output exit code in FAILED message #1805

Open

Felixoid marked this pull request as draft December 3, 2024 16:36

Felixoid force-pushed the master branch from 91bffbe to 2eb867f Compare December 3, 2024 18:05

Felixoid marked this pull request as ready for review December 3, 2024 18:08

Felixoid mentioned this pull request Dec 3, 2024

Ninja hides the exit status of failing subprocesses #1507

Open

Felixoid force-pushed the master branch from 8bd3571 to 0f56f38 Compare December 3, 2024 21:26

Felixoid force-pushed the master branch 7 times, most recently from 24bca44 to 5579fc5 Compare December 9, 2024 23:36