Limit sample size for the pod collector

[0x2b3bfa0]

Desired behavior

It would be nice to have an option to limit the sample size (i.e. number of sampled pods) for the pod collector:

metric-config.pods.queue-size.json-path/max-pod-sample-size: 1

Limiting the sample size to 1 will cause the pod collector to sample only one pod in the deployment (e.g. at random, or the first responsive one) and use the returned metric, instead of sampling all the pods.

This is useful when pods expose metrics that aren't pod-scoped, but come from a central database or queue system.

Equivalent workaround

Create a Kubernetes Service named e.g. example for the pods exposing the metrics endpoint, and then use it with the external metric collector like this:

      metric-config.external.datachain-queued-jobs.json-path/endpoint: http://example.studio.svc.cluster.local/metrics

[0x2b3bfa0]

I've written an example implementation:

• Add max-pod-sample-size annotation for the pod collector #744

If there's any interest on upstreaming this, I'll open a polished pull request for review.