Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Kubernetes Provider] Only store necessary metadata for Jobs #5788

Open
MichaelKatsoulis opened this issue Oct 15, 2024 · 0 comments
Open

[Kubernetes Provider] Only store necessary metadata for Jobs #5788

MichaelKatsoulis opened this issue Oct 15, 2024 · 0 comments

Comments

@MichaelKatsoulis
Copy link
Contributor

This issue is a continuation of #5623.

Kubernetes Provider (and add_kubernetes_metadata processor) if configured with the following block:

      add_resource_metadata:
        cronjob: true

add the kubernetes.cronjob.name metadata field to a log event of a pod that is handled by a job, which is handled by a cronJob.
This behaviour was by default disabled after a certain stack version because it can lead to increase in elastic-agent's memory consumption.
This happens in environments with many cronJobs that generated a lot of ephemeral pods. The API requests are increased but also the data stored in memory are increased as well.

The addition of the cronJob name is achieved by creating a new watcher-informer mechanism for Jobs. Then the cronJob name is extracted by the Job object and added in pod log event.
The code as it is right now, stores the whole Job object in memory although we just need its name and the cronJob name that created it, if it exists.

As a follow up of the work done for replicaSets we can use the same mechanism of only keeping the necessary data for the Job resource.

Related Issues:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant