rust: Parallelize build better by starting next Rust target already once metadata (rmeta) of all dependencies is available

[sdroege]

Describe the bug

rustc is currently called by meson to only produce an rlib or dylib via --emit link. Another kind of output rustc can produce is metadata.

The metadata contains everything that is needed for compiling the next targets that depend on this one, and cargo uses this to parallelize the build better. Basically, you can already start compiling the next target while LLVM is doing its things (optimizer, etc) on the previous one still.

In case of cargo this is done by enabling the JSON status output of rustc, which then notifies once the metadata is available or the linking is done. Theoretically both steps can also be done independently by calling rustc twice but that will be slower as a lot of work has to be duplicated between both runs.

It's not clear to me how this could be integrated into meson but @nirbheek said he has some ideas.

CC @dcbaker

[nirbheek]

(sorry, re-commenting, I accidentally posted the previous one incomplete)

The idea I had requires a wrapper command implemented in C (since python's startup cost is too high). The wrapper command will have three modes:

1. "invoked to generate metadata" aka metadata mode
2. "invoked to generate rlib" aka rlib mode
3. "invoked to run rustc" aka rustc mode (this is only needed to be able to get the stderr output and exit code of rustc)

What we'd do is:

• Define two targets: one that outputs a metadata file (wrapper metadata mode), and one that outputs an rlib (wrapper rlib mode)
• Both targets will actually be generated by running a single rustc command underneath (wrapper rustc mode)
• The rlib target will depend on the metadata target, ensuring that the metadata target is always run first.
• The metadata target, when invoked, will run the wrapper command in metadata mode which will spawn (or fork) itself as a subprocess in rustc mode which will run rustc as a subprocess and write its own PID to a pre-defined file.
• The process will exit immediately when it gets notified over stdout that the metadata file has been written, and allow the subprocess in rustc mode to continue. Ninja will treat this as a successful generation of the target and move on.
• When ninja invokes the rlib target, it will run the wrapper command in rlib mode which will wait for the PID written to the file to exit, and return the same error code that it returns (communicated via another file by the wrapper running in rustc mode)

[xclaesse]

I'm not sure to understand how it works with cargo, please correct me if I got it wrong:

• There is a single rustc invocation.
• At some point in the middle of rustc invocation, it creates a json file.
• Cargo watch for the creation of the json file and start next compilation.

In theory rustc invocation could be split in 2 invocations, the first stops after creating the json file and the 2nd do the actual compilation/link, but in practice that 2nd invocation redo too much of the processing already done in 1st invocation?

So we need 2 ninja targets, one for rlib and one for json.

• rlib: that can be a direct rustc invocation as we do right now.
• json: that needs to be a thin wrapper that just wait for json file to appear. That could be inotify, but that's not portable so we need to build some IPC between 2 processes: 1 that read rustc's stdout and tell the 2nd one when the json is ready.

[sdroege]

Almost. The JSON part is parseable stdout status reporting from rustc, which could tell us when the metadata file is available so we don't have to do inotify or similar. The metadata file itself is a binary file.

[xclaesse]

That means that for every rustc process we need to spawn 3 other processes, that's still significant overhead especially on Windows, no?

I guess the alternative is adding in ninja support for a process to notify ninja that some dep files are already available before the process exit.

[xclaesse]

I'm currious if C++20 modules has a similar situation.

[eli-schwartz]

Theoretically both steps can also be done independently by calling rustc twice but that will be slower as a lot of work has to be duplicated between both runs.

What's the practical effect of this? How much would we actually be paying to offset the increased parallelism opportunity?

BTW it would be great if rustc had an option to share this information :/ something vaguely like a precompiled header or, with the way rust seems to be headed, a background daemon.

Which is what this proposal essentially is, isn't it? A background daemon but implemented with flag files instead of the process table.

[nirbheek]

for every rustc process we need to spawn 3 other processes

I believe this is only for the link stage, so it's 3 extra processes per link.

[sdroege]

I believe this is only for the link stage, so it's 3 extra processes per link.

There's only a link stage for Rust targets (unless you want to split it into depinfo, metadata, link and duplicate work between them).

[dcbaker]

Theoretically both steps can also be done independently by calling rustc twice but that will be slower as a lot of work has to be duplicated between both runs.

I mean, we can't do this because rustc can't read in rmeta for the crate under compilation, only for other crates. I thought there was talk about supporting this though from the rustc side.

There's only a link stage for Rust targets (unless you want to split it into depinfo, metadata, link and duplicate work between them).

We've actually had requests to support having rmeta and object files separate, to avoid linking the stdlib into every rlib, which would make the actualy in-meson implementation a lot cleaner (and will likely be needed to support gcc-rs). We could implement this today, since an rlib is (currently) just an archive with the rmeta as the first file, and then the object file(s) after that.

[sdroege]

@xclaesse has some WIP code for this at ninja-build/ninja#2287 and #11707 .

For my test project this reduces compilation time by 8% with buildtype=debug and by 15% with buildtype=debugoptimized, which is something. Using opt-level=3 will probably reduce it by something like 20%.

This makes sense because we'd start compiling the next target while LLVM is busy, and optimizations are mostly done in LLVM and taking up time there.