Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

integration-test: Set rust-lld as a linker only on macOS #908

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

vadorovsky
Copy link
Member

@vadorovsky vadorovsky commented Mar 13, 2024

The recommendation (coming from rust-lang/rust#130062) for Linux hosts
is using C compiler driver as a linker, which is able to find
system-wide libraries. Using linker binaries directly in -C linker
(e.g. -C linker=rust-lld) often results in errors like:

cargo:warning=error: linking with `rust-lld` failed: exit status: 1ger, ppv-lite86, libc...
cargo:warning=  |
cargo:warning=  = note: LC_ALL="C" PATH="/home/vadorovsky/.rustup/toolchains/stable-x86_64-un
cargo:warning=  = note: rust-lld: error: unable to find library -lgcc_s
cargo:warning=          rust-lld: error: unable to find library -lc
cargo:warning=
cargo:warning=
cargo:warning=
cargo:warning=error: aborting due to 1 previous error

Not touching the linker settings is usually the best approach for Linux
systems. Native builds pick up the default C toolchain. Cross builds
default to GCC cross wrapper, but that's easy to supress with clang
and lld using RUSTFLAGS.

However, -C linker=rust-lld still works the best on macOS, where
Rust toolchains come with unwinder and runtime and there is usually no
need to link system libraries. Keep setting it only for macOS.

Fixes #907

Copy link

netlify bot commented Mar 13, 2024

Deploy Preview for aya-rs-docs ready!

Built without sensitive environment variables

Name Link
🔨 Latest commit ae94156
🔍 Latest deploy log https://app.netlify.com/sites/aya-rs-docs/deploys/670a9d44775793000820e15d
😎 Deploy Preview https://deploy-preview-908--aya-rs-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@mergify mergify bot added the test A PR that improves test cases or CI label Mar 13, 2024
@vadorovsky vadorovsky changed the title integration-test: Prefer system-wide lld over rust-lld integration-test: Prefer system-wide lld over rust-lld on Linux Sep 7, 2024
@vadorovsky vadorovsky force-pushed the integration-tests-linker branch 3 times, most recently from c3a1ecf to 0107a99 Compare September 7, 2024 18:13
@tamird
Copy link
Member

tamird commented Oct 7, 2024

Is this no longer necessary?

@vadorovsky
Copy link
Member Author

vadorovsky commented Oct 8, 2024

Is this no longer necessary?

I would say it's still necessary. I'm still facing the same issue when I'm building integration tests on any musl-based system.

I asked Rust community about that here rust-lang/rust#130062 and the conclusion is basically:

"you're supposed to use the C compiler driver as the linker, as it knows where to find these libraries"

It's still a mystery for me why -C linker=rust-lld doesn't face the same issue on Ubuntu and I'm only able to reproduce that on Alpine and Gentoo(musl-llvm) with *-musl toolchains, but their explanation makes sense. I ended working around this issue on my side with custom RUSTFLAGS (-C linker=clang -C link-arg=-fuse-ld=lld) and commenting the chunk of code that this PR touches.

Let me rebase it. The red builds are still the old ones from September, unrelated to the PR itself.

@vadorovsky
Copy link
Member Author

vadorovsky commented Oct 8, 2024

On a side note, Rust nightly is currently using rust-lld by default (but still through the system C compiler which figures out the paths). If they decide to ship that in any stable version, we won't have to touch any RUSTFLAGS here anymore.

@vadorovsky
Copy link
Member Author

vadorovsky commented Oct 8, 2024

Ubuntu fails now because of old clang. I also noticed just now that we are entering the if let Some(target) statement even for native builds, when running integration tests...

Taking a step back, I'm starting to doubt whether we should mangle with the linkers and linker flags at all in this place. Our default cargo configuration specifies the cross gcc wrappers as linkers, which should be good enough for Ubuntu/Debian/Fedora users even for cross-compilation scenarios. People wanting to link with clang+lld could just define their own RUSTFLAGS (at least I'm fine with doing that,). I will give it a try now, but if not touching the flags is going to work both on Ubuntu and macOS, then I think that would be a solution.

@tamird
Copy link
Member

tamird commented Oct 9, 2024

Ubuntu fails now because of old clang. I also noticed just now that we are entering the if let Some(target) statement even for native builds, when running integration tests...

That doesn't sound right. Are you sure?

Taking a step back, I'm starting to doubt whether we should mangle with the linkers and linker flags at all in this place. Our default cargo configuration specifies the cross gcc wrappers as linkers, which should be good enough for Ubuntu/Debian/Fedora users even for cross-compilation scenarios. People wanting to link with clang+lld could just define their own RUSTFLAGS (at least I'm fine with doing that,). I will give it a try now, but if not touching the flags is going to work both on Ubuntu and macOS, then I think that would be a solution.

How is aarch64-linux-musl-gcc going to work for someone running on macOS? I believe I added that for exactly this reason.

@vadorovsky
Copy link
Member Author

vadorovsky commented Oct 9, 2024

Ubuntu fails now because of old clang. I also noticed just now that we are entering the if let Some(target) statement even for native builds, when running integration tests...

That doesn't sound right. Are you sure?

Nevermind, that's not true. cargo xtask integration-test local without any --target flag doesn't enter the statement and works fine on musl distros.

What doesn't work is building/running VM tests (on musl/Linux host). In that case target is set, rust-lld used as a linker and it's struggling with finding the libs.

$ cargo xtask integration-test vm gentoo-kernel-6.6.51-1/image/usr/src/linux-6.6.51-gentoo-dist/arch/x86/boot/bzImage
[...]
cargo:warning=  = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "rust-lld" "-flavor" "gnu" "--version-script=/tmp/rustcpfJE43/list" "--no-undefined-version" "/tmp/rustcpfJE43/symbols.o" "/src/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.0.rcgu.o" "/src/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.1.rcgu.o" "/src/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.2.rcgu.o" "/src/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.3.rcgu.o" "/src/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.4.rcgu.o" "/src/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.5.rcgu.o" "/src/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.6.rcgu.o" "/src/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.7.rcgu.o" "/src/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.8.rcgu.o" "/src/target/debug/deps/rustversion-a51ee28261058709.f01o1e0hk9d1mcr47j33gnmo6.rcgu.rmeta" "/src/target/debug/deps/rustversion-a51ee28261058709.8219qqgkv2oj66wv6vzyrwhri.rcgu.o" "--as-needed" "-L" "/src/target/debug/deps" "-L" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Bstatic" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libproc_macro-d2aa9b2c4004c068.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-1b60b176e45a0318.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-57b45c66d2e62c49.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libobject-0123438d987de009.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libmemchr-05f3c42b49d78619.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libaddr2line-45c700404e36c3de.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libgimli-cf5f9b37b545ed6e.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-44abbf672f21a10b.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_detect-9b6d712a985955e4.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-e4ec61de4b8ef447.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-c72777eb88217fe3.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libminiz_oxide-b0a8429443e0fed6.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libadler-820055e0e9cc55a4.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-2ceb129e8b1d5c77.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-8cd2ce184499d012.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-ada011f709b467ef.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-67a0423d44ab989a.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-15fc71c4e52ac90f.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-273f69939f389095.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-c8f82c72237d07e8.rlib" "-Bdynamic" "-lgcc_s" "-lc" "--eh-frame-hdr" "-z" "noexecstack" "-L" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-o" "/src/target/debug/deps/librustversion-a51ee28261058709.so" "--gc-sections" "-shared" "-soname=librustversion-a51ee28261058709.so" "-z" "relro" "-z" "now"
cargo:warning=  = note: rust-lld: error: unable to find library -lgcc_s
cargo:warning=          rust-lld: error: unable to find library -lc
cargo:warning=
cargo:warning=
cargo:warning=
cargo:warning=error: aborting due to 1 previous errorures-util, hashbrown, libc, clap_builder, winnow, libc, quote, rustversion, r...
cargo:warning=
cargo:warning=
error: could not compile `rustversion` (lib) due to 2 previous errors

After the following change:

diff --git a/xtask/src/run.rs b/xtask/src/run.rs
index 8bba7a5..b6e4b2a 100644
--- a/xtask/src/run.rs
+++ b/xtask/src/run.rs
@@ -59,8 +59,7 @@ where
     let mut cmd = Command::new("cargo");
     cmd.args(["build", "--message-format=json"]);
     if let Some(target) = target {
-        let config = format!("target.{target}.linker = \"rust-lld\"");
-        cmd.args(["--target", target, "--config", &config]);
+        cmd.args(["--target", target]);
     }
     f(&mut cmd);

It works. At least for the x86_64 scenario, where my default C toolchain is picked up. For cross scenarios, I would need to set up RUSTFLAGS manually to pick up clang+lld, but it's fine. I just don't want xtask automatically switching to rust-lld in that case.

I also checked whether that change works on Ubuntu and it also works fine. Using C compiler as linker indeed looks like the best approach for Linux.

How is aarch64-linux-musl-gcc going to work for someone running on macOS? I believe I added that for exactly this reason.

I see. I guess that -C linker=rust-lld works on macOS, because unlike on Linux, rustup ships the unwinder and runtime libraries in together with the toolchain and then the linker doesn't try to find them in the system.

I tried whether the combination of -C linker=rust-lld -C link-args=-fuse-ld=lld works on macOS, but nope, it throws the following:

-contained/crtn.o"
cargo:warning=  = note: ld64.lld: error: unknown argument '--as-needed'
cargo:warning=          ld64.lld: error: unknown argument '-Bstatic', did you mean '-static'
cargo:warning=          ld64.lld: error: unknown argument '-Bdynamic', did you mean '-dynamic'
cargo:warning=          ld64.lld: error: unknown argument '--eh-frame-hdr'
cargo:warning=          ld64.lld: error: unknown argument '-z'
cargo:warning=          ld64.lld: error: unknown argument '--gc-sections'
cargo:warning=          ld64.lld: error: unknown argument '-z'
cargo:warning=          ld64.lld: error: unknown argument '-z'
cargo:warning=          ld64.lld: error: unknown argument '--strip-debug'
cargo:warning=          clang: error: linker command failed with exit code 1 (use -v to see invocation)
cargo:warning=
cargo:warning=
cargo:warning=
cargo:warning=error: aborting due to 1 previous error
cargo:warning=
cargo:warning=
error: could not compile `init` (bin "init") due to 2 previous errors
Error: building init program failed
Caused by:
    "cargo" "build" "--message-format=json" "--target" "aarch64-unknown-linux-musl" "--target" "aarch64-unknown-linux-musl" "--config" "target.aarch64-unknown-linux-musl.linker = \"clang\"" "--config" "target.aarch64-unknown-linux-musl.rustflags = \"-C link-arg=-fuse-ld=lld\"" "--package" "init" "--profile" "release" failed: ExitStatus(unix_wait_status(25856))

So I guess that we could apply -C linker=rust-lld only on macOS host - hacky, but I think that will make everyone happy.

The recommendation (coming from rust-lang/rust#130062) for Linux hosts
is using C compiler driver as a linker, which is able to find
system-wide libraries. Using linker binaries directly in `-C linker`
(e.g. `-C linker=rust-lld`) often results in errors like:

```
cargo:warning=error: linking with `rust-lld` failed: exit status: 1ger, ppv-lite86, libc...
cargo:warning=  |
cargo:warning=  = note: LC_ALL="C" PATH="/home/vadorovsky/.rustup/toolchains/stable-x86_64-un
cargo:warning=  = note: rust-lld: error: unable to find library -lgcc_s
cargo:warning=          rust-lld: error: unable to find library -lc
cargo:warning=
cargo:warning=
cargo:warning=
cargo:warning=error: aborting due to 1 previous error
```

Not touching the linker settings is usually the best approach for Linux
systems. Native builds pick up the default C toolchain. Cross builds
default to GCC cross wrapper, but that's easy to supress with clang
and lld using RUSTFLAGS.

However, `-C linker=rust-lld` still works the best on macOS, where
Rust toolchains come with unwinder and runtime and there is usually no
need to link system libraries. Keep setting it only for macOS.

Fixes aya-rs#907
@vadorovsky vadorovsky changed the title integration-test: Prefer system-wide lld over rust-lld on Linux integration-test: Set rust-lld as a linker only on macOS Oct 9, 2024
@tamird
Copy link
Member

tamird commented Oct 9, 2024

Thanks for the detail. Let's give it a shot. Can we add a CI job that verifies that everything works as expected?

@vadorovsky vadorovsky force-pushed the integration-tests-linker branch 3 times, most recently from 6c0bf0e to 18aa37c Compare October 12, 2024 15:08
This way we are making sure that the integration tests infra doesn't
regress on musl environments.
@vadorovsky vadorovsky force-pushed the integration-tests-linker branch 2 times, most recently from 8175be3 to ae94156 Compare October 12, 2024 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test A PR that improves test cases or CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

integration-test: rust-lld invoked by cargo xtask integration-test vm fails to find libraries
2 participants