Add building and testing of PGO+LTO LLVM and Julia #345

Zentrik · 2024-03-26T11:10:57Z

I ended up just using the profile from Julia being built for PGO, locally it seems to work fine. If anyone has better ideas for what to profile, please do say.

I don't think we need to run the full test suite but it seems good to run at least some stuff to make sure nothing got messed up. The testing seems to finish a fair bit quicker than the gnuassertrr tests and not that far behind the other test pipelines, so this shouldn't slow down CI much.

Zentrik · 2024-03-26T11:22:29Z

I don't think I have permissions to fix ERROR: Pipeline '.buildkite/pipelines/main/launch_upload_jobs.yml' fails treehash signature check! You may need to re-run cryptic/bin/sign_treehashes!.

Zentrik · 2024-03-26T14:34:21Z

Build currently failing as the bump to patchelf JuliaLang/julia#53845 introduced a check that the compiler supports C++ 17. Not sure why that check is failing on CI as it does seem to be using clang 16.

We run /cache/build/builder-amdci4-3/julialang/julia-buildkite/deps/srccache/patchelf-0.18.0//configure --prefix=/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage1.build/usr --build=x86_64-unknown-linux-gnu --libdir=/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage1.build/usr/lib --bindir=/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage1.build/usr/tools LDFLAGS="-fuse-ld=lld -flto=thin -fprofile-generate=/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/profiles -Wl,-rpath,'\$\$ORIGIN' -Wl,-z,origin -Wl,-rpath-link,/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage1.build/usr/lib -Wl,--enable-new-dtags " F77="gfortran -m64" CC="/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang " CXX="/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang++ " LD="/cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/ld.lld" LDFLAGS="" CPPFLAGS="" for the PGO build and the failure is checking whether /cache/build/builder-amdci4-3/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang++ supports C++17 features with -std=c++17... no.
For the non PGO build we run /cache/build/tester-amdci5-9/julialang/julia-buildkite/deps/srccache/patchelf-0.18.0//configure --prefix=/cache/build/tester-amdci5-9/julialang/julia-buildkite/usr --build=x86_64-linux-gnu --libdir=/cache/build/tester-amdci5-9/julialang/julia-buildkite/usr/lib --bindir=/cache/build/tester-amdci5-9/julialang/julia-buildkite/usr/tools LDFLAGS=" -Wl,-rpath,'\$\$ORIGIN' -Wl,-z,origin -Wl,-rpath-link,/cache/build/tester-amdci5-9/julialang/julia-buildkite/usr/lib -Wl,--enable-new-dtags " F77="gfortran -m64" CC="gcc -m64 " CXX="g++ -m64 " LD="ld" LDFLAGS="" CPPFLAGS="" and checking whether g++ -m64 supports C++17 features with -std=c++17... yes is the relevant check.

The line that runs the check is https://github.com/NixOS/patchelf/blob/99c24238981b7b1084313aca8f5c493bb46f302c/configure.ac#L35 and the macro defined in https://github.com/NixOS/patchelf/blob/master/m4/ax_cxx_compile_stdcxx.m4.

The only obvious difference I see is the -m64 but I really have no clue why this is failing on CI.

Zentrik · 2024-03-26T19:39:35Z

Maybe clang is picking the wrong gcc?

Zentrik · 2024-03-26T20:36:47Z

Ok so the actual error is

configure:6320: /cache/build/builder-amdci4-6/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang++ -m64  -std=c++17 -c -fprofile-generate=/cache/build/builder-amdci4-6/julialang/julia-buildkite/contrib/pgo-lto/profiles -Xclang -mllvm -Xclang -vp-counters-per-site=6  conftest.cpp >&5
conftest.cpp:437:10: fatal error: 'initializer_list' file not found
#include <initializer_list>

JuliaLang/julia#45641 (comment) mentions something about libstdc++ detection, so maybe something to do with that.
Clang does seem to be loading /usr/lib/gcc/x86_64-linux-gnu/10 the only gcc installation it finds whereas under 'Print software versions' we see gcc version 9.

haampie · 2024-03-26T21:13:03Z

Is that a partial gcc 10 install? No g++10 / libstdc++-10-dev?

You can force clang to use gcc 9 if necessary with --gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/9

Notice that --gcc-toolchain is entirely useless when multiple GCCs are installed in the same prefix /usr.

Zentrik · 2024-03-26T21:31:53Z

I'm quite out of my depth here but I'm not sure gcc is installed in /usr/lib/gcc/x86_64-linux-gnu/9 because this was printed which seems to suggest it's installed elsewhere

COLLECT_LTO_WRAPPER=/usr/local/bin/../libexec/gcc/x86_64-linux-gnu/9.1.0/lto-wrapper
Target: x86_64-linux-gnu
Configured with: /workspace/srcdir/gcc-9.1.0/configure --prefix=/workspace/destdir --with-build-sysroot=/workspace/destdir/x86_64-linux-gnu --with-sysroot=/workspace/destdir/x86_64-linux-gnu --with-gxx-include-dir=/workspace/destdir/x86_64-linux-gnu/include/c++/9.1.0 --target=x86_64-linux-gnu --host=x86_64-linux-gnu --build=x86_64-linux-gnu --disable-multilib --disable-werror --disable-bootstrap --enable-threads=posix --program-prefix=x86_64-linux-gnu- --with-arch=x86-64 --enable-languages=c,c++

Zentrik · 2024-03-26T21:34:56Z

It does seem to find the correct gcc version now. Yes, it does seem to get through patchelf's configure phase but now fails at making it

/cache/build/builder-amdci4-0/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang++ -DPACKAGE_NAME=\"patchelf\" -DPACKAGE_TARNAME=\"patchelf\" -DPACKAGE_VERSION=\"0.18.0\" -DPACKAGE_STRING=\"patchelf\ 0.18.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"patchelf\" -DVERSION=\"0.18.0\" -DHAVE_CXX17=1 -I. -I/cache/build/builder-amdci4-0/julialang/julia-buildkite/deps/srccache/patchelf-0.18.0/src    -Wall -Wextra -Wcast-qual -std=c++17 -D_FILE_OFFSET_BITS=64     -fprofile-generate=/cache/build/builder-amdci4-0/julialang/julia-buildkite/contrib/pgo-lto/profiles -Xclang -mllvm -Xclang -vp-counters-per-site=6 -MT patchelf.o -MD -MP -MF .deps/patchelf.Tpo -c -o patchelf.o /cache/build/builder-amdci4-0/julialang/julia-buildkite/deps/srccache/patchelf-0.18.0/src/patchelf.cc
/cache/build/builder-amdci4-0/julialang/julia-buildkite/deps/srccache/patchelf-0.18.0/src/patchelf.cc:19:10: fatal error: 'algorithm' file not found
#include <algorithm>

haampie · 2024-03-26T21:35:40Z

The build environment likely contains just the C compiler part of GCC 10, and clang has poor detection & heuristics for the GCC install dir. If multiple are in the same prefix it picks the newest, and it will do so also when there are missing components like C++ stdlib header files. The --gcc-toolchain flag is useles cause it expects the prefix /usr, in which both GCC 9 and 10 live.

Use --gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/9 or uninstall GCC 10.

Zentrik · 2024-03-26T21:44:59Z

/cache/build/builder-amdci5-3/julialang/julia-buildkite/contrib/pgo-lto/stage0.build/usr/tools/clang  -fprofile-generate=/cache/build/builder-amdci5-3/julialang/julia-buildkite/contrib/pgo-lto/profiles -Xclang -mllvm -Xclang -vp-counters-per-site=6 -O2 -m64 --gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/9 -fPIC -std=c99 -Wsign-conversion -Wall -Wextra -pedantic -DUTF8PROC_EXPORTS  -c -o utf8proc.o utf8proc.c
clang-16: error: '/usr/lib/gcc/x86_64-linux-gnu/9' does not contain a GCC installation

On the previous commit I saw

Found candidate GCC installation: /usr/local/bin/../lib/gcc/x86_64-linux-gnu/9.1.0//../../../../lib/gcc/x86_64-linux-gnu/9.1.0
Selected GCC installation: /usr/local/bin/../lib/gcc/x86_64-linux-gnu/9.1.0//../../../../lib/gcc/x86_64-linux-gnu/9.1.0

So I presume the specified path was wrong.

Zentrik · 2024-03-26T22:03:54Z

Thanks, it seems to be working much better now. I just need to figure out why there's a hash mismatch, I had the same locally but I just refreshed it.

pipelines/main/platforms/build_linux.pgo_lto.arches

Zentrik · 2024-04-21T17:32:24Z

This looks good to me, testing was 15% faster. If we want -Werror I think we just need to merge or otherwise fix JuliaLang/julia#53897 and bump libwhich to vtjnash/libwhich@613b3f0.

pipelines/main/launch_unsigned_jobs.yml

Somewhat hacky way to fix JuliaCI#314 and to get JuliaCI#345 working again.

* Update CMake version used when building Somewhat hacky way to fix #314 and to get #345 working again. * Only update CMake with llvm_passes image * Potential Fix * Try print debugging * Fix typo * Remove debugging echos

giordano · 2024-09-20T23:49:32Z

I discussed this with @Zentrik a few days ago: I believe it'd be great to actually test the PGO+LTO pipeline, but I'm a bit concerned with the extra long job this introduces. One option could be to move the job as a scheduled one, with the counter effect few eyes will look at it and the risk this will bitrot at some point. What do people think?

Zentrik · 2024-09-21T08:57:49Z

In terms of speeding up the build job we could build the instrumented LLVM on yggy. Further, we could only profile periodically, e.g. every LLVM upgrade, allowing us to skip the slow profiling step and to distribute the optimised LLVM from yggy.

giordano reviewed Mar 29, 2024

View reviewed changes

pipelines/main/platforms/build_linux.pgo_lto.arches Outdated Show resolved Hide resolved

Zentrik force-pushed the main branch 4 times, most recently from f2ffa42 to 5ebf7b5 Compare April 21, 2024 12:17

Zentrik commented Jul 13, 2024

View reviewed changes

pipelines/main/launch_unsigned_jobs.yml Show resolved Hide resolved

Zentrik commented Jul 13, 2024

View reviewed changes

pipelines/main/launch_unsigned_jobs.yml Outdated Show resolved Hide resolved

Zentrik force-pushed the main branch from 2698cec to 20bdb46 Compare July 26, 2024 17:51

Zentrik added a commit to Zentrik/julia-buildkite that referenced this pull request Jul 27, 2024

Update CMake version used when building

10347f4

Somewhat hacky way to fix JuliaCI#314 and to get JuliaCI#345 working again.

Zentrik mentioned this pull request Jul 27, 2024

Update CMake version used when building #370

Merged

Zentrik force-pushed the main branch from 20bdb46 to 9c55562 Compare July 28, 2024 07:03

Zentrik mentioned this pull request Jul 28, 2024

Building LLVM on CI seems broken after Deps: after configuring CMake project, use cmake --build, not make ( #54538) JuliaLang/julia#55284

Closed

Zentrik force-pushed the main branch from 29e3c83 to 9b28b0b Compare August 7, 2024 18:56

Zentrik changed the title ~~Add building and testing of PGO+LTO LLVM and Julia~~ Add building and testing of PGO+LTO+BOLT LLVM and Julia Aug 17, 2024

Zentrik marked this pull request as draft August 19, 2024 06:32

Zentrik changed the title ~~Add building and testing of PGO+LTO+BOLT LLVM and Julia~~ Add building and testing of PGO+LTO LLVM and Julia Aug 22, 2024

Zentrik force-pushed the main branch 2 times, most recently from 94f7c00 to a0b0a3b Compare August 25, 2024 07:47

Zentrik and others added 16 commits September 9, 2024 21:28

Try to prevent binary-dist from rebuilding sysimg.ji

62cebaf

Try again

eaabb50

Try again

05efb04

Try again

df41626

Try again

63b21a3

Try again

214bb07

Test fix

0de5f46

Try again

4b7099c

Fix again

aa3e34c

Retrigger

ea9a02f

Try again

482f04a

Try again

d165dc5

Fix again

6595ba2

Fix up

8dc52bd

Rebuild

b4a1a2f

Switch back to official Julia Repo

faa645d

Zentrik force-pushed the main branch from 6d34338 to faa645d Compare September 9, 2024 20:28

Zentrik marked this pull request as ready for review September 9, 2024 20:28

Zentrik marked this pull request as draft September 9, 2024 21:43

Zentrik added 4 commits September 10, 2024 20:15

Try fix for lld not being installed into usr/tools

a6d823d

Hopefully correct fix

a51739b

Retrigger

173d95e

Switch back to Julia repo

399ae7e

Zentrik marked this pull request as ready for review September 11, 2024 18:05

Merge branch 'main' into main

acca367

DilumAluthge requested review from giordano, staticfloat and gbaraldi September 20, 2024 23:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add building and testing of PGO+LTO LLVM and Julia #345

Add building and testing of PGO+LTO LLVM and Julia #345

Zentrik commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024 •

edited

Loading

haampie commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024

haampie commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024

Zentrik commented Apr 21, 2024

giordano commented Sep 20, 2024

Zentrik commented Sep 21, 2024

Add building and testing of PGO+LTO LLVM and Julia #345

Are you sure you want to change the base?

Add building and testing of PGO+LTO LLVM and Julia #345

Conversation

Zentrik commented Mar 26, 2024 • edited Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024 • edited Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024 • edited Loading

haampie commented Mar 26, 2024 • edited Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024

haampie commented Mar 26, 2024 • edited Loading

Zentrik commented Mar 26, 2024

Zentrik commented Mar 26, 2024

Zentrik commented Apr 21, 2024

giordano commented Sep 20, 2024

Zentrik commented Sep 21, 2024

Zentrik commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024 •

edited

Loading

Zentrik commented Mar 26, 2024 •

edited

Loading

haampie commented Mar 26, 2024 •

edited

Loading

haampie commented Mar 26, 2024 •

edited

Loading