[Failing Test]: Firestore integration tests failing affecting GCPIO_PreCommit, DataflowPostCommit V1 and V2

[Abacn]

What happened?

Test Link: https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_DataflowV2/lastCompletedBuild/testReport/
More Info:

Many FirestoreIO integration test in the Apache Beam Java PostCommit failing starting March 14 without code change.

Error Message
java.util.concurrent.ExecutionException: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Invalid request routing header database=projects/apache-beam-testing/databases/firestoredb. Please fill in the request header with format x-goog-request-params:project_id=apache-beam-testing&database_id=firestoredb

Firestore recently enforced the routing header for named databases. Integration tests need to pass the correct routing header.

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[kennknowles]

We need to schedule the work to make tests that work. I am unclear whether we need this at P1. I guess we lost a lot of coverage...

[Abacn]

Most test are back green and executing again in #26738, except one still failing, waiting for another firestore release.