Error setting up Splunk OpenTelemetry Collector for Kubernetes

[oarusoft]

I am trying to setup OpenTelemetry Collector for Kubernetes in Splunk Cloud.

For this I followed the following article

Splunk OpenTelemetry Collector for Kubernetes

I am using google Kubernetes engine. As mentioned in article I tried both below two helm to install Splunk OpenTelemetry Collector in a Kubernetes cluster. None of these worked and I keep getting error message " no such host".

helm install my-splunk-otel-collector --set="splunkPlatform.endpoint=https://http-inputs-prd-p-19lmt.splunkcloud.com:8088/services/collector/event,splunkPlatform.token=XXXX-XXXX-XXXX-XXXX-XXXX,splunkPlatform.index=main,clusterName=spluk-kub" splunk-otel-collector-chart/splunk-otel-collector

helm install my-splunk-otel-collector --set="splunkPlatform.endpoint=https://prd-p-19lmt.splunkcloud.com:8088/services/collector/event,splunkPlatform.token=XXXX-XXXX-XXXX-XXXX-XXXX,splunkPlatform.index=main,clusterName=spluk-kub" splunk-otel-collector-chart/splunk-otel-collector

Here is how the error log loooks like

2022-09-01T04:43:43.953Z info exporterhelper/queued_retry.go:427 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post "[https://http-inputs-prd-p-19lmt.splunkcloud.com:8088/services/collector/event](https://http-inputs-prd-p-19lmt.splunkcloud.com:8088/services/collector/event/)": dial tcp: lookup http-inputs-prd-p-19lmt.splunkcloud.com on 10.4.0.10:53: no such host", "interval": "32.444167474s"}
2022-09-01T04:43:52.374Z error exporterhelper/queued_retry.go:176 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "max elapsed time expired Post "[https://http-inputs-prd-p-19lmt.splunkcloud.com:8088/services/collector/event](https://http-inputs-prd-p-19lmt.splunkcloud.com:8088/services/collector/event/)": dial tcp: lookup http-inputs-prd-p-19lmt.splunkcloud.com on 10.4.0.10:53: no such host", "dropped_items": 1}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure

Please advice . Thank you.

[atoulme]

This error indicates that your collector cannot connect to the https://http-inputs-prd-p-19lmt.splunkcloud.com:8088/services/collector/event endpoint.

Please ensure this endpoint is valid and responds. Currently this host is not registered under a DNS record:

$> host http-inputs-prd-p-19lmt.splunkcloud.com                              
Host http-inputs-prd-p-19lmt.splunkcloud.com not found: 3(NXDOMAIN)

Please refer to this documentation to establish the correct endpoint associated with your Splunk Cloud instance: https://docs.splunk.com/Documentation/Splunk/9.0.1/Data/UsetheHTTPEventCollector#Send_data_to_HTTP_Event_Collector_on_Splunk_Cloud_Platform

[oarusoft]

I did followed that articate before and built my endpoint.

I am using Splunk Cloud trail and my Splunk cloud website is https://prd-p-19lmt.splunkcloud.com/

Based on the docment the sysntax is ://http-inputs-.splunkcloud.com:/

I modified a bit to use raw instead of event and still getting error "no such host"
Below is what I used.

helm install my-splunk-otel-collector --set="splunkPlatform.endpoint=https://http-inputs-prd-p-19lmt.splunkcloud.com:8088/services/collector/raw,splunkPlatform.token=xxxxxxxxx,splunkPlatform.index=main,clusterName=splunk" splunk-otel-collector-chart/splunk-otel-collector
NAME: my-splunk-otel-collector

Please suggest what should I use when tail website is https://prd-p-19lmt.splunkcloud.com/

Thanks

[atoulme]

It looks like your host doesn't exist:

$> host prd-p-9vxw3.splunkcloud.com
Host prd-p-9vxw3.splunkcloud.com not found: 3(NXDOMAIN)

Are you sure this host name is correct?

[cgreengr]

I think I found the error. Thank you so much!

[cgreengr]

@atoulme

[vinitmasaun]

@atoulme

We are experiencing a similar error but it doesn't say no such host and we verified that the Otel collector pod is able to communicate with the splunkhec endpoint. Following is the error printed in the Otel-collector container:

2023-05-02T20:47:19.458Z    error    exporterhelper/queued_retry.go:175    Exporting failed. No more retries left. Dropping data.    {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "max elapsed time expired Post "https://<removed_from_public_view>:8088/services/collector": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)", "dropped_items": 133}

We verified the hec token is correct and are able to use curl with the same token to send test event to the hec endpoint from the collector pod. No logs are being indexed from the k8s cluster due to the above error. We have changed the log level to debug but we couldn't find any additional clue in the debug log statements. What else could be the cause of this issue?

[atoulme]

I am asking our support team to reach out and work with you to examine what is happening. They will ask you questions, if they don't have the information already, around which version of the helm chart and collector you are running, what you are connecting to, etc. I can recommend that you try out v0.76.0 with us if you like as it has performance fixes that can help. We would want to debug the Splunk endpoint as well, especially if you go direct to ingest or go through a HF. Let's continue this discussion directly through support.

[karanbp24]

Getting same error:
2024-08-15T15:38:08.975Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post "https://http-inputs-prd-p-1ypci.splunkcloud.com:8088/services/collector/raw\": dial tcp: lookup http-inputs-prd-p-1ypci.splunkcloud.com on 172.20.0.10:53: no such host", "interval": "6.110259216s"}
as per documentation its supposed to have http-inputs before host name.
Not sure if the documentation has misinformation or I am following is wrong

[ihari03]

I am using Splunk Enterprise not cloud.