[Task]: Improve how to handle the Dataflow-specific option impersonateServiceAccount for Beam Java

[liferoad]

What needs to happen?

impersonateServiceAccount should be kept when submitting Dataflow jobs but should be removed when creating Dataflow workers per the design. To fix this, #30283 put a simple solution to remove the impersonateServiceAccount key from the JSON pipeline options. This introduces some Dataflow-specific concepts, which could be improved by moving it to the Dataflow-specific module. See more details in this comment.

Open this issue to track this potential task to improve how to handle Dataflow-specific options in the future.

Note for Beam Python, we remove this option from the internal Dataflow apiclient module

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[kennknowles]

For this particular option, the dataflow service (the UW) should be the place where you remove the option.

The Python SDK is a real mess when it comes to isolating non-GCP and GCP things. It is not a good place to use as an example.