Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OTEL enrichment agent do not set labels or host metadata during scaling event #1275

Open
rupeshnemade opened this issue Oct 6, 2023 · 2 comments

Comments

@rupeshnemade
Copy link

We observed that whenever OTEL pod scales up we start seeing few logs with incomplete metadata (labels, host).
Expected
image

Actual
As we can see there are no labels & host present in Kuberenetes block
image

During scaling phase as OTEL pod doesn't have metadata from k8s API in its cache it sends logs without setting few metadata fields.
This behaviour is valid for short-lived pods but not appropriate for running pods.

Is this Kubernetes problem or this can be fixed in OTEL?

@swiatekm
Copy link

swiatekm commented Oct 9, 2023

Since this only happens for new otel Pods, I have a suspicion that it's because the k8sprocessor processed data without first having populated its Pod cache completely. As I suggested on Slack, can you try adding a delay to the otel container (for example via initialDelaySeconds in the readiness probe) and see if that helps?

If this is the case, we can consider explicitly waiting for caches to sync while starting the processor.

@rupeshnemade
Copy link
Author

rupeshnemade commented Oct 9, 2023

I tried updating initialDelaySeconds in OTEL pods forwarding logs to Sumo from default 5sec to 20 sec but it didn't help to alleviate the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants