ROX-26272 Enable ccache for faster builds #1838

robbycochran · 2024-09-10T21:40:18Z

Description

Use ccache for local and remote ansible builds for both builder libraries and collector binaries.
Optimized s390x git checkout by disabling recursive submodule checkout
Settings: ccache enabled by default on all CI jobs. It is off by default locally. It can be disabled with no-ccache label on PRs.

Total speed ~3x comparing recent runs for Main Collector CI (~3-4hrs to ~1.5hr). The bulk coming from the very slow buildx builds. The running times below are from representative builds (not avg) and show the time for the the entire job, including set up and teardown.

Without ccache With ccache Speedup

amd64 builder 14m 55s 7m 45s 1.92x

arm64 builder 2h 22m 35m 13s 4.03x

ppc64 builder 2h 33m 37m 55s 4.04x

s390x builder 31m 41s 14m 57s 2.12x

amd64 6m 22s 2m 38s 2.42x

arm64 52m 14s 7m 32s 6.93x

ppc64 50m 52s 8m 37s 5.90x

s390x 22m 39s 10m 52s 2.08x

Prior art from @Molter73

Checklist

Investigated and inspected CI test results
Updated documentation accordingly

Automated testing

Added unit tests
Added integration tests
Added regression tests

Testing Performed

Inspected CI.

Molter73

Just some minor comments, but overall looking great!

Molter73 · 2024-09-17T07:44:19Z

builder/install/install-dependencies.sh

+    # create a wrapper utility for clang-17 used by modern bpf builds in falco
+    printf '#!/bin/sh\nexec ccache /usr/bin/clang-17 "$@"\n' > /usr/local/bin/ccache-clang
+    chmod +x /usr/local/bin/ccache-clang
+    echo /usr/local/bin/ccache-clang


Side note, after we pull this change we shouldn't need this wrapper anymore: falcosecurity/libs#2032

Nice! I'll add a note to remind us to remove

builder/install/80-libbpf.sh

Molter73 · 2024-09-17T07:55:53Z

builder/install/ccache.sh

+    git clone https://github.com/ccache/ccache.git third_party/ccache
+
+    cd third_party/ccache
+    git fetch --tags
+    git checkout "${CCACHE_VERSION}"


We could use a submodule checked out to CCACHE_VERSION, but since it won't be used for konflux or downstream, this is fine as well.

Yeah I made the same calculation. I don't feel that strongly about it.

builder/install/ccache.sh

Molter73 · 2024-09-17T08:16:35Z

.github/workflows/collector-builder.yml

+      - name: Set up builder ccache in docker cache
+        if: env.USE_CCACHE && matrix.arch != 's390x'
+        uses: reproducible-containers/[email protected]
+        with:
+          cache-map: |
+            {
+              "${{ github.workspace }}/builder/.ccache": "/root/.ccache"
+            }


Since you had to implement the logic that this action provides for s390x, I wonder if it makes sense to just handle all archs the same way in our ansible playbooks and drop this step, mostly for consistency, this approach is fine.

Molter73 · 2024-09-17T08:20:39Z

.github/workflows/collector-builder.yml

@@ -17,6 +17,7 @@ env:
  COLLECTOR_TAG: ${{ inputs.collector-tag }}
  DEFAULT_BUILDER_TAG: master
  ANSIBLE_CONFIG: ${{ github.workspace }}/ansible/ansible.cfg
+  USE_CCACHE: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ccache') }}


Do we want to run with ccache on the master branch and nightlies? That way PRs could just pull the cache from there and see a speed up from the very first commit that gets pushed, otherwise the first build will be as slow as the ones we have now. Maybe we can check if the branch being used is master somewhere in our build and clean the cache before building? That way the master branch would always be built from a clean slate, but the cache would still be provided for PRs.

Suggested change

USE_CCACHE: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ccache') }}

USE_CCACHE: ${{ github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'no-ccache') }}

https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#example-of-search-priority

Ah good idea. We could skip cache restore on master but always save the cache.

0116bf7 ccache 7fe4a87 typo 4ba08ca ansible format 31fb575 add .ccache to .gitignore f5e5fd3 ccache on builder image 4c75f3a debug logs for builder 3412c94 create logs 29778a0 typo b22b325 print 3ba1265 key a414bb1 remove debug d12f47b enable s390x caching 5578333 typo 10b4a40 s390x 9c4ffa1 skip docker inject action on s390x 66d4e7f fix path 4a58a89 builtin f16c110 fix deda8ba format 23d5ed4 var ef8e17a order 65b1816 fmt 2378492 simplify 89c69c6 scripts 4170881 version 1b191f8 path be6e04b task time b8f61ba quote 30382e5 cache file for remote vm b8c982b fmt 15c70af var a2f148b update a927675 check dir a77bd93 enable builder, s390x collector 0ba7d0a typo 664e31a dbg 28ace91 broken cache f822219 ls 9d5ccdf rm 8ce9e57 slash e80f8f5 slash 1482ea1 permissions and fix builder ccache 3fb4a1a s390x 4ee116e s390x 0887d1d root 6bb2700 dbg b9e8ed1 Revert "dbg" ab60dff rm opts c85e21e collector bdb810b no ccache avail

optimize git on s390x

robbycochran requested a review from a team as a code owner September 10, 2024 21:40

robbycochran added build-builder-image run-multiarch-builds Run steps for non-x86 archs. labels Sep 10, 2024

Base automatically changed from giles/pin-ansible-version-2.16 to master September 11, 2024 11:09

robbycochran force-pushed the rc-ccache branch 2 times, most recently from bdb810b to 2231b2f Compare September 14, 2024 17:32

robbycochran changed the title ~~[testing] ccache~~ Enable ccache for faster builds Sep 16, 2024

robbycochran force-pushed the rc-ccache branch from 6e4c460 to 17ba285 Compare September 16, 2024 22:13

robbycochran changed the title ~~Enable ccache for faster builds~~ ROX-26272 Enable ccache for faster builds Sep 17, 2024

Molter73 reviewed Sep 17, 2024

View reviewed changes

robbycochran added 12 commits September 27, 2024 16:06

X-Smart-Branch-Parent: origin/master

d441fd3

create ccache dir on remote for collector build

5479a24

optimize git on s390x

key names

6366dc3

fmt

1693602

key

b8980b7

label

770f8c3

fmt

72a9257

optional steps

56fe902

debug ansible

abdf93a

fmt

7f13405

fix

4f69acf

robbycochran force-pushed the rc-ccache branch from 5e622ca to 4f69acf Compare September 27, 2024 23:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ROX-26272 Enable ccache for faster builds #1838

ROX-26272 Enable ccache for faster builds #1838

robbycochran commented Sep 10, 2024 •

edited

Loading

Molter73 left a comment

Molter73 Sep 17, 2024

robbycochran Sep 17, 2024

Molter73 Sep 17, 2024

robbycochran Sep 17, 2024

Molter73 Sep 17, 2024

Molter73 Sep 17, 2024

robbycochran Sep 17, 2024

	Without ccache	With ccache	Speedup
amd64 builder	14m 55s	7m 45s	1.92x
arm64 builder	2h 22m	35m 13s	4.03x
ppc64 builder	2h 33m	37m 55s	4.04x
s390x builder	31m 41s	14m 57s	2.12x
amd64	6m 22s	2m 38s	2.42x
arm64	52m 14s	7m 32s	6.93x
ppc64	50m 52s	8m 37s	5.90x
s390x	22m 39s	10m 52s	2.08x

	USE_CCACHE: ${{ !contains(github.event.pull_request.labels.*.name, 'no-ccache') }}
	USE_CCACHE: ${{ github.event_name != 'pull_request' \|\| !contains(github.event.pull_request.labels.*.name, 'no-ccache') }}

ROX-26272 Enable ccache for faster builds #1838

Are you sure you want to change the base?

ROX-26272 Enable ccache for faster builds #1838

Conversation

robbycochran commented Sep 10, 2024 • edited Loading

Description

Checklist

Testing Performed

Molter73 left a comment

Choose a reason for hiding this comment

Molter73 Sep 17, 2024

Choose a reason for hiding this comment

robbycochran Sep 17, 2024

Choose a reason for hiding this comment

Molter73 Sep 17, 2024

Choose a reason for hiding this comment

robbycochran Sep 17, 2024

Choose a reason for hiding this comment

Molter73 Sep 17, 2024

Choose a reason for hiding this comment

Molter73 Sep 17, 2024

Choose a reason for hiding this comment

robbycochran Sep 17, 2024

Choose a reason for hiding this comment

robbycochran commented Sep 10, 2024 •

edited

Loading