[Bug]: Github Action build_release_candidate publish_java_artifacts job fails with GC overhead limit exceeded.

[lostluck]

What happened?

The build_release_candidate workflow is not currently usable to actually build release candidates.

This occurs when running the build_release_candidate workflow publish_java_artifacts job:

* What went wrong:
Execution failed for task ':sdks:java:io:pulsar:compileJava'.
> A build operation failed.
      GC overhead limit exceeded
   > GC overhead limit exceeded

This happened twice, for different gradle tasks. I'm resorting to local runs for the 2.50.0 release RC1 at least, as a result.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[lostluck]

cc: @damccorm for triage.

[damccorm]

This looks like an issue with the task itself (maybe a memory leak). Let me know how the local run goes, I'd kinda expect the same outcome there though.

[damccorm]

Interestingly, this happened once in solr io - https://github.com/apache/beam/actions/runs/5872911524/job/15925292759
and once in pulsar io - https://github.com/apache/beam/actions/runs/5881177706/job/15949111373, so its not a consistent failure.

[damccorm]

One thing we could try here is changing the workflow to run on our self hosted runners (I'd thought it already was) - https://github.com/apache/beam/blob/master/.github/workflows/build_release_candidate.yml which might have more space available? We could also create a pool with more memory for this purpose.

Lets see how local run goes before jumping to that though

[lostluck]

Interestingly, this happened once in solr io - https://github.com/apache/beam/actions/runs/5872911524/job/15925292759 and once in pulsar io - https://github.com/apache/beam/actions/runs/5881177706/job/15949111373, so its not a consistent failure.

I did say This happened twice, for different gradle tasks. I was not able to replicate the issue locally with either task.

[lostluck]

As a practical matter, if it's a workflow change I can point the action to, I'm happy to try that out instead of locally.

My laptop is currently busy with the docker pushes that I won't interrupt, and I'm happy to try whatever in the mean time.

Laptop = Serial.

[lostluck]

Last I recall, we removed all the "self-hosted" runners though because they didn't work. I gather that's changed then?

[damccorm]

Yeah, we're in the process of migrating all workflows which started with stabilizing our self hosted runners