Gradle 8 support

[Abacn]

Fixes #28127

• Make gradle help --scan green

• Replicate defunct com.palantir.docker and docker-run plugins

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[codecov]

Codecov Report

Merging #28756 (ed5aa63) into master (6f4e285) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #28756      +/-   ##
==========================================
- Coverage   72.21%   72.21%   -0.01%     
==========================================
  Files         684      684              
  Lines      101226   101230       +4     
==========================================
+ Hits        73103    73104       +1     
  Misses      26547    26547              
- Partials     1576     1579       +3     

Flag	Coverage Δ
go	53.41% <ø> (+<0.01%)	⬆️
python	82.76% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 13 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[kennknowles]

Now that 2.51.0 release branch is cut, this should be OK to merge.

[Abacn]

We have to vendor the following gradle plugins

• com.palantir.docker -> BeamDockerPlugin
• com.palantir.docker-run -> BeamDockerRunPlugin

Due to

• v0.34.0 used gradle internal API that incompatible with gradle 8
• v0.35.0 requries Java11; recompile from src does not work either because the plugin depends on other palantir dependencies which also requires Java11
• palantir/gradle-docker is now marked as Disclaimer: This Repo is now Defunct, "although we will keep it working, no new features are accepted"

We can choose another plugin, but it would require substantial change to the code base, and the risk of support stopped on any third party plugin remains

If palantir/gradle-docker still get maintained and beam eventually drops Java 8 support, we can still come back to this plugin.

Fortunately beam only uses moderate functionality in palantir/gradle-docker so vendor is straightforward. The artifact provides 3 plugins

• com.palantir.docker
• com.palantir.docker-run
• com.palantir.docker-compose

beam only uses the first two. The third is the most complicated one and depends on gradle internal APIs; in docker plugin, there is a dockerfileZip task also involves gradle internal API. Beam uses neither - and they are not included in this PR

The two vendored plugins here then get rid of gradle internal APIs and work with Gradle 8 while keep build with Java 8.

[kennknowles]

Thank you for the explanation. These plugins seem fairly trivial and it might even be useful to just have our own. Otherwise we could file bugs to migrate to some well-supported plugin later.

[Abacn]

the latest commit just fixes comments and causing hadoop io precommit failing. The reason is gradle cache corrupted: https://stackoverflow.com/questions/65645510/cannot-access-script-base-class-org-gradle-kotlin-dsl-kotlinbuildscript

[Abacn]

Now that 2.51.0 release branch is cut, this should be OK to merge.

Thanks, marked this as ready for review

R: @kennknowles @bvolpato

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[kennknowles]

Thank you for the explanation. These plugins seem fairly trivial and it might even be useful to just have our own. Otherwise we could file bugs to migrate to some well-supported plugin later.

[Abacn]

Let me run some java11/17 postcommits before merge

[Abacn]

Run Jpms Direct Java 11 PostCommit

[Abacn]

Run Jpms Flink Java 11 PostCommit

[Abacn]

Run Jpms Flink Java 17 PostCommit

[Abacn]

Run Jpms Direct Java 17 PostCommit

[Abacn]

Run Java 17 Examples on Dataflow Runner V2

[Abacn]

Run Java 11 Examples on Dataflow Runner V2

[Abacn]

Run Java examples on Dataflow Java 17

[Abacn]

Run Java examples on Dataflow Java 11

[kennknowles]

The whitespace precommit appears to be failing: https://github.com/apache/beam/actions/runs/6415209350

Because it does things that will be forbidden in Gradle 9.