S3 and high usage of GetObject

[evilr00t]

Hi!

After setting up Thanos and moving existing metrics into Thanos & S3 we've noticed high activity of GetObject on our S3 bucket (Data on thanos 300GiB, data on S3 Bucket ~1TiB as we have versioning)

Cost for GetObject for last 24hrs is 10-25USD/day which seems to be pretty high. Using Server Access Logs I was able to identify that StoreGateway and Compactor are 2 main services doing GetObject, are there any settings that could lower amount of requests to S3?

~/thanos-0.35.0.darwin-amd64 ./thanos tools bucket inspect --objstore.config-file=bucket.yaml | wc -l
ts=2024-06-10T12:07:02.641904Z caller=factory.go:53 level=info msg="loading bucket configuration"
ts=2024-06-10T12:07:03.309642Z caller=fetcher.go:626 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=666.439625ms duration_ms=666 cached=76 returned=76 partial=0
ts=2024-06-10T12:07:03.319353Z caller=main.go:174 level=info msg=exiting
      78

Thanos Version: latest (0.35.1)

[evilr00t]

After increasing/adjusting in-memory caching this is not an issue anymore.

Maybe it would be worth it to set in-memory cache by default to some % value from assigned memory? Default value seems to be useless.