[Bug]: Kafka integration test log spam

[Abacn]

Related Template(s)

N/A

Template Version

N/A

What happened?

Java PR Action has long been flaky, at least in the past it clearly shows which tests failed. However, recently the test log size has increased substantially, likely due to Kafka template and test development. Now the log is of > 80 MB.

A majority of the log reads

2024-08-19T15:39:29.6622952Z [kafka-producer-network-thread | producer-7] INFO org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-7] Node 1 disconnected.
2024-08-19T15:39:29.6626237Z [kafka-producer-network-thread | producer-7] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-7] Connection to node 1 (/10.128.0.40:57609) could not be established. Node may not be available.
2024-08-19T15:39:29.6629592Z [kafka-admin-client-thread | adminclient-3] INFO org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-3] Node 1 disconnected.
2024-08-19T15:39:29.6633503Z [kafka-admin-client-thread | adminclient-3] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-3] Connection to node 1 (/10.128.0.40:57609) could not be established. Node may not be available.

and

2024-08-19T13:31:21.7130918Z [docker-java-stream--996043679] INFO org.apache.beam.it.testcontainers.TestContainerResourceManager - confluentinc/cp-kafka:7.3.1: [2024-08-19 13:23:47,251] INFO [Broker id=1] Handling LeaderAndIsr request correlationId 1 from controller 1 for 5 partitions (state.change.logger)
2024-08-19T13:31:21.7133326Z 
2024-08-19T13:31:21.7143323Z [docker-java-stream--996043679] INFO org.apache.beam.it.testcontainers.TestContainerResourceManager - confluentinc/cp-kafka:7.3.1: [2024-08-19 13:23:47,253] TRACE [Broker id=1] Received LeaderAndIsr request LeaderAndIsrPartitionState(topicName='testkafkatogcsbinaryencoding-20240819-132347-044185', partitionIndex=0, controllerEpoch=1, leader=1, leaderEpoch=0, isr=[1], partitionEpoch=0, replicas=[1], addingReplicas=[], removingReplicas=[], isNew=true, leaderRecoveryState=0) correlation id 1 from controller 1 epoch 1 (state.change.logger)

Relevant log output

No response

[AnandInguva]

I think we need to separate the Kafka IT tests from Java tests since there are so many Kafka IT tests. Having a separate GH action for kafka tests. would solve this partially

[AnandInguva]

I can take a look at this later this week and put up a PR that separates the Kafka tests into a different GH actions suite.

[Abacn]

The log spam causes the mvn command stuck for 30 minutes to process logs to console, between the last test run and finalize, see #1817 (comment)

We have to disable printing logs until resolved.

[AnandInguva]

@Abacn John mentioned that there is a way to configure Kafka to emit fewer logs.

We have to disable printing logs until resolved.
What do you mean by this?

[Abacn]

@Abacn John mentioned that there is a way to configure Kafka to emit fewer logs.

We have to disable printing logs until resolved.
What do you mean by this?

Like replace "-e" to "-q" here:

DataflowTemplates/cicd/internal/op/maven.go

Line 32 in 3b59345

fullArgs = append(fullArgs, "-e")

to see if the workflow no longer stuck at processing log

[Abacn]

Try to set loglevel but did not work

LogManager logManager = LogManager.getLogManager();
  java.util.logging.Logger rootLogger = logManager.getLogger("");
  rootLogger.setLevel(Level.WARNING);

or

java.util.logging.Logger logger = java.util.logging.Logger.getLogger("org.apache.beam.it.testcontainers.TestContainerResourceManager");
logger.setLevel(Level.WARNING);

likely because it was added as a log consumer to test container:

https://github.com/apache/beam/blob/6901d7c862388ded58e1cda3286c429edab58c7c/it/testcontainers/src/main/java/org/apache/beam/it/testcontainers/TestContainerResourceManager.java#L75

[Abacn]

The original issue title noted two separate issue (Java test flaky / Kafka log spam), now Kafka PR has been separated from Java PR, though the log spam issue still present. Changed the title and assigned to kafka test owner