Observability Kubernetes Onboarding doesn't ship data

[flash1293]

Following the Kubernetes onboarding flow on serverless (Add data > Monitor Infrastructure > Kubernetes) doesn't ship data. This can be reproduced on a serverless observability project and was tested with minikube running on Mac.

The logs show lots of errors like this:

{"log.level":"error","@timestamp":"2024-09-25T08:15:35.683Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed kubernetes/metrics-default-kubernetes-node-metrics-kubernetes-bcdc9c26-d274-4db3-95e0-0bb396fdd402 (STARTING->FAILED): Failed: pid '295' exited with code '-1'","log":{"source":"elastic-agent"},"component":{"id":"kubernetes/metrics-default","state":"FAILED"},"unit":{"id":"kubernetes/metrics-default-kubernetes-node-metrics-kubernetes-bcdc9c26-d274-4db3-95e0-0bb396fdd402","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}

It's possible this is a problem on the Kibana side in the flow as well, starting here for troubleshooting and we can move the issue in case it's unrelated.

A suspicion is that this is related to resourcing and the agent now needs more memory, but this needs to be confirmed.

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[MichaelKatsoulis]

This log error does not say anything about the reason it crashed. We would need to reproduce the environment and check the diagnostics and the agent pod consumption

[flash1293]

@MichaelKatsoulis

I started a local minikube cluster, then followed the onboarding flow from a fresh Observability serverless project on prod.

[MichaelKatsoulis]

I replicated the scenario:

1. Kind cluster with 38 pods and 3 nodes
2. Fresh serverless project

I followed the instruction of monitoring Kubernetes as if I was a first time user.

I noticed the following:

1. The kustomize command attempts to override the Elasticsearch host by setting

-e "s/%ES_HOST%/https:\/\/katsoulis-serverless-f68892.es.us-east-1.aws.elastic.cloud/g"

Elastic-Agent in the absence of a port, appends the port in the end which by default is 9200. So the ES_HOST ends up https://katsoulis-serverless-f68892.es.us-east-1.aws.elastic.cloud:9200
This leads to connection refused. In order to overcome this, we need to modify the ES_HOST to

-e "s/%ES_HOST%/https:\/\/katsoulis-serverless-f68892.es.us-east-1.aws.elastic.cloud:443/g"

2. Elastic-Agent starts successfully and data are flowing
   

3. The first thing a user sees is a link to a dashboard which does not exist!
   

4. In discovery we can see metrics and logs
   

5. After some minutes we see the first restart of one of the agent's pods.
   

6. Reason is OOM killed
   

Conclusion:

Restarts:
As per my analysis and tests in #4729 (comment)
in version 8.15.1 elastic-agent with Kubernetes and system integration needs more than 700Mb of memory.
So the limit is set low causing restarts.

Dashboard
Should also Kubernetes Integration be installed under the hood which contains the assets?

ES_HOST
We should always set the port of Elasticsearch because if not set, agent appends 9200.

[flash1293]

Thanks for the investigation @MichaelKatsoulis !

Restarts:
As per my analysis and tests in #4729 (comment)
in version 8.15.1 elastic-agent with Kubernetes and system integration needs more than 700Mb of memory.
So the limit is set low causing restarts.

I guess this is something that needs to be changed on the elastic-agent side, right?

Should also Kubernetes Integration be installed under the hood which contains the assets?

Good catch, seems like the id of the dashboard changed in this PR: elastic/integrations#10593 We should fix it short-term, but we need to think how we can make this whole process more stable.

We should always set the port of Elasticsearch because if not set, agent appends 9200.

I see, I think in a previous version it would append it, but the config value we pull this from changed. We can fix this on the Kibana side as well.