HTTP 404 for Splunk OpenTelemetry Collector for Kubernetes

[oarusoft]

This is related to https://github.com/signalfx/splunk-otel-collector-chart/discussions/526
I created this seprate disussion since that issue was resolved.

I tried diffrent below helm commands for prod and getting the below error. In prod the Kubernetes is installed on AWS (EKS)

---

helm install my-splunk-otel-collector --set="splunkPlatform.endpoint=https://http-inputs-example-dev.splunkclould.com:8088/services/collector/event,splunkPlatform.token=XXXX,splunkPlatform.index=mulesoft_poc,clusterName=mulesoft-rtf-sandbox-eks-us-e1-1,splunkPlatform.insecureSkipVerify=true" splunk-otel-collector-chart/splunk-otel-collector

***Error Log>>>

2022-09-02T21:16:06.472Z info service/collector.go:128 Everything is ready. Begin running and processing data.
2022-09-02T21:16:24.497Z info exporterhelper/queued_retry.go:427 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post "https://http-inputs-example-dev.splunkclould.com:8088/services/collector/event\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "4.577287637s"}

---

helm install my-splunk-otel-collector --set="splunkPlatform.endpoint=https://http-inputs.example-dev.splunkclould.com/services/collector/event,splunkPlatform.token=XXXX,splunkPlatform.index=mulesoft_poc,clusterName=mulesoft-rtf-sandbox-eks-us-e1-1" splunk-otel-collector-chart/splunk-otel-collector

***Error Log>>>

2022-09-02T21:08:47.368Z error exporterhelper/queued_retry.go:395 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Permanent error: "HTTP/1.1 400 Bad Request\r\nContent-Length: 44\r\nConnection: keep-alive\r\nContent-Type: application/json; charset=UTF-8\r\nDate: Fri, 02 Sep 2022 21:08:47 GMT\r\nServer: Splunkd\r\nVary: Authorization\r\nX-Content-Type-Options: nosniff\r\nX-Frame-Options: SAMEORIGIN\r\n\r\n{\"text\":\"Data channel is missing\",\"code\":10}"", "dropped_items": 2}

---

helm install my-splunk-otel-collector --set="splunkPlatform.endpoint=https://example-dev.splunkclould.com/services/collector/event,splunkPlatform.token=XXXX,splunkPlatform.index=mulesoft_poc,clusterName=mulesoft-rtf-sandbox-eks-us-e1-1,splunkPlatform.insecureSkipVerify=true" splunk-otel-collector-chart/splunk-otel-collector

***Error Log>>>

2022-09-02T17:02:21.914Z info exporterhelper/queued_retry.go:427 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "HTTP 404 "Not Found"", "interval": "8.564041535s"}

---

Please advice
Thank You

[oarusoft]

I was able to fix this error. The problem was not sending the channel token.

[LisaParrattBJSS]

It'd be nice if you gave some sort of indication how you went about making it send a channel token, because it's not obvious.