Exporting very recent metrics (before tsdb blocks) of promethues HA

[santoshmungle]

Hi,

I have prometheus HA with Thanos deployed as a kubernetes cluster. I would like to export very recent metrics of prometheus HA (2 prom instances) including deduplication to remote location for some analytic use cases.

I see that there is a "Thanos Receiver" component that addresses this. However, it replaces "Sidecar" as "Sidecar" is not necessary when you use "Receiver" according to the documentation. In this case, I will have to change all my kubernetes manifests.

Is there any other alternative you can suggest? So that I don't have to change kubernetes manifests significantly.

Thank you.

[bwplotka]

Yes.

You could use the snapshot technique and "flush" blocks to object storage manually. See more here: prometheus-junkyard/tsdb#346

We never prepared anything automated for it, but we are happy to guide someone to do it (: Help wanted 🤗

[bwplotka]

What I am saying: If there would be a thanos tools blocks flush CLI that you can run in the docker exec and it will do the work.. Tha would work for you?

[santoshmungle]

That might do the trick. However, what about deduplication? If I am flushing WAL of two prom instances to remote location, I would still face issue of deduplication as WAL of two prom instances will have duplicated metric for analytic use cases.

[bwplotka]

Well that is the same as if you don't flush and wait 2h no? You can run store + querier on such data that will deduplicate data for you.

Other option is run batch jobs that will deduplicate the blocks (compactor)... when this is done #1014 🤗

[santoshmungle]

In that case, can we push thanos-query result (metric resulted from PromQL queries) as querier deduplicate metric to remote location?

In my case, for example, I would have custom developed dashboard (not Grafana) hosted on remote location with specific analytic use cases requiring metric from specific PromQL queries executed through thanos-query HTTP endpoint.