KafkaSource error: the timeout period of 600000ms has been exceeded while executing POST

[ashrafguitoni]

Expected Behavior

No error logs even on long-running requests

Actual Behavior

$ kubectl -n knative-eventing logs -f kafka-source-dispatcher-0 shows logs like:

{
  "@timestamp": "2024-07-22T07:03:39.584Z",
  "@version": "1",
  "message": "failed to send event to subscriber context={topics=[my-requests], consumerGroup='my-workflow-group', reference=uuid: \"d256b565-5054-4343-8003-c2821259f127\"\nnamespace: \"default\"\nname: \"my-workflow\"\n} target=http://my-workflow.default.svc.cluster.local",
  "logger_name": "dev.knative.eventing.kafka.broker.dispatcher.impl.http.WebClientCloudEventSender",
  "thread_name": "vert.x-worker-thread-8",
  "level": "ERROR",
  "level_value": 40000,
  "stack_trace": "io.vertx.core.http.impl.NoStackTraceTimeoutException: The timeout period of 600000ms has been exceeded while executing POST  for server my-workflow.default.svc.cluster.local:80\n",
  "context": {},
  "target": "http://my-workflow.default.svc.cluster.local"
}

Steps to Reproduce the Problem

As far as I can tell, all that it takes is to run a request via a kafkasource requires a long time to process (our requests can take up to 20 minutes)

Our kafkasource looks like this:

apiVersion: sources.knative.dev/v1beta1
kind: KafkaSource
metadata:
  name: my-workflow
  namespace: default
spec:
  bootstrapServers: ["kafka.default.svc.cluster.local:9092"]
  consumerGroup: my-workflow-group
  consumers: 1
  initialOffset: latest
  ordering: ordered
  sink:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: my-workflow
      namespace: default
  topics: ["my-requests"]

Additional Info

• I'm not sure if this is really a bug, but I'd like to know if there's a way to get rid of this error.
• I'm using Knative serving and Knative eventing (including the kafka source controller) versions 1.12