[Task]: Add test scenarios that verify that Python SDK remains not affected by releases of Beam's dependencies.

[tvalentyn]

What needs to happen?

The goal is to check that:

1. the latest Python SDK in PyPI is remains installable and usable if tomorrow some Beam's dependency promotes their currently available release candidate, to be a release. This suite should install apache-beam[gcp] from pypi, instead of using Beam's SDK tarball (dev SDK). Additionally, this test should install apache beam with --pre flag https://pip.pypa.io/en/stable/cli/pip_install/#install-pre.

2. Same as 1), but use Dev SDK, and install the new dependencies in Beam's docker containers. One way to add this scenario is to reuse a Dataflow Python ValidatesContainer test, but install Beam with --pre flag.

3. Verify that installing Beam HEAD doesn't involve dependency resolution backtracking. We may have to disable this signal if it becomes unactionable or hard to enforce, but seems like something we can catch and prevent.

To make test signals easier to debug, it would be good to have a tool that can easily tell us which changes in dependency chain have recently happen. Most likely there is already something that can be reused.

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

cc: @riteshghorse

[tvalentyn]

part of #25652

[AnandInguva]

.take-issue

[tvalentyn]

We should also take a look at which dependencies have RC releaseses, and which don't and suggest having RCs to respective maintainers

[tvalentyn]

Verify that installing Beam HEAD doesn't involve dependency resolution backtracking.

Decided to not do this at this time.

We should also take a look at which dependencies have RC releaseses, and which don't and suggest having RCs to respective maintainers

Reached out to GCP client library maintainers. Can do this as needed if certain dependencies cause issues for us.