Memcached timeout related question

[edgarkz]

Hi,
1.Can you please assist with tempo queries logging the following timeouts from memcached?
Looking into memcached metrics I don't see anything hitting max connections or so.

2.should we scale up more instances of memcached? the chart supports it but will be any benefit from multiple instances?
single memcached instance deployed from distributed tempo helm chart

eks 1.26
tempo 2.4.1 distributed via helm chart

level=error ts=2024-03-28T21:20:22.175685654Z caller=memcached.go:153 msg="Failed to get keys from memcached" err="memcache: connect timeout to 10.100.213.119:11211" level=error ts=2024-03-28T21:20:22.175931644Z caller=memcached.go:153 msg="Failed to get keys from memcached" err="memcache: connect timeout to 10.100.213.119:11211" level=error ts=2024-03-28T21:20:22.434598693Z caller=memcached.go:236 msg="failed to put to memcached" name=parquet-footer|bloom|frontend-search err="server=10.100.213.119:11211: memcache: connect timeout to 10.100.213.119:11211" level=error ts=2024-03-28T21:20:23.133903161Z caller=memcached.go:236 msg="failed to put to memcached" name=parquet-footer|bloom|frontend-search err="server=10.100.213.119:11211: memcache: connect timeout to 10.100.213.119:11211" level=error ts=2024-03-28T21:20:24.174768798Z caller=memcached.go:236 msg="failed to put to memcached" name=parquet-footer|bloom|frontend-search err="server=10.100.213.119:11211: memcache: connect timeout to 10.100.213.119:11211" level=error ts=2024-03-28T21:20:24.175104043Z caller=memcached.go:236 msg="failed to put to memcached" name=parquet-footer|bloom|frontend-search err="server=10.100.213.119:11211: memcache: connect timeout to 10.100.213.119:11211" level=error ts=2024-03-28T21:20:24.195413573Z caller=memcached.go:236 msg="failed to put to memcached" name=parquet-footer|bloom|frontend-search err="server=10.100.213.119:11211: memcache: connect timeout to 10.100.213.119:11211" level=error ts=2024-03-28T21:20:24.195483153Z caller=memcached.go:236 msg="failed to put to memcached" name=parquet-footer|bloom|frontend-search err="server=10.100.213.119:11211: memcache: connect timeout to 10.100.213.119:11211" level=error ts=2024-03-28T21:20:24.195515305Z caller=memcached.go:236 msg="failed to put to memcached" name=parquet-footer|bloom|frontend-search err="server=10.100.213.119:11211: memcache: connect timeout to 10.100.213.119:1121

[joe-elliott]

We see similar logs when a memcached cluster becomes overwhelmed and generally respond by scaling up the cluster. Sizing and operating memcached on large clusters is still something we're working out so I'd love to compare notes.

One thing that can help is separating your cache configs. I notice that your logs indicate that you're using one cache cluster and all roles are configured together (parquet-footer|bloom|frontend-search). If you keep one cluster but simply break the config apart then Tempo will metric and log them independently which may help us determine if this is related to one of the roles.

cache:
    caches:
        - memcached:
            host: <some cluster>
          roles:
            - bloom
            - parquet-footer
            - ...

becomes

cache:
    caches:
        - memcached:
            host: <some cluster>
          roles:
            - bloom
        - memcached:
            host: <some cluster>
          roles:
            - parquet-footer
        - ...

I'll also point out that we set max_idle_conns to 100 on all clusters. Honestly not sure how tuned that is. Like I said, we are exploring this space right now.

We also set connections on our memcached servers: -c 4096 and monitor them by watching the memcached_current_connections metric.

@electron0zero has a bit more experience here and my have insights.

[edgarkz]

thank you @joe-elliott, I'll do some tests in staging and reply back.

[electron0zero]

@edgarkz as @joe-elliott said, we are also seeing some of these errors and figuring out the sizing things.

here are few general heuristics I use to know when to scale a cache:

• Look at cache latency, and if cache latency is hitting cache_timeout, that means that cache is under scaled and it’s taking too long to read or write, scale the cache.
• If a cache has a high eviction rate, cache might be under provisioned, scale the cache.
• Lower level cache like bloom cache, parquet-page cache, parquet-footer cache will see higher hit rate (usually around 90% of above), if you have a consistent query traffic and these lower level caches have low hit rate, they are undervalued and needs to be scaled up.
• Higher level cache like frontend-search cache will have a low hit rate and will only be useful when the same query is being repeated. Size these according to the amount of data you want you want to cache

if you try these, let me know how it goes :)

[chenlujjj]

Hi @electron0zero @joe-elliott , I've met a similar issue:
The querier component logs show lots of "Failed to get keys from memcached"

The monitor of memcached is like:

As you can see, there is no evicts here.

We have 4 replicas of memcached, the memory limit is 8 GiB, the extra args include:

- --conn-limit=4096  # default is 1024
- --memory-limit=7168  # default is 64 MB

What configurations do you recommend to tune for memcached to resolve this? Thanks

[edgarkz]

Hey @electron0zero @joe-elliott,
Sorry for late update - we decided to move away from builtin memcached instance and using amazon ElastiCache OOS instead
I have configured 2 different endpoints per usage:
caches:
- roles:
- parquet-footer
- frontend-search
redis:
endpoint: xxxxxx
- roles:
- bloom
redis: xxxxxxxxx

All the connection related issues are gone away and we have in place monitoring for scaling now.
Can you please advise if we should enable any additional caching on redis? some basic guidelines will be much appreciated

"caches available to storage backend" footer=true bloom=true offset_idx=false column_idx=false trace_id_idx=false page=false

[edgarkz]

thanks @electron0zero
You have mentioned more roles but i don't see in docs anything about :
offset_idx=false column_idx=false trace_id_idx=false page=false
2nd question - regarding bloom cache - i see the memory usage only increases for this specific cache:

is it intentional? the above is 1 day stats.
Eventually i'll hit the memory limit if it continues

[electron0zero]

offset_idx=false column_idx=false trace_id_idx=false page=false

this is a just a difference in the name of roles in the log line, compared to config values, let me make them same as config names.

Eventually i'll hit the memory limit if it continues

yes, tempo expects the cache to do the eviction. from what I know, you can configure the eviction policy in redis to take care of this.

[electron0zero]

log line is updated in #4005

[edgarkz]

@electron0zero can you please advise if you set any TTL on cache objects?
by default Redis OOS on amazon uses the below eviction policy:

The cache evicts the least recently used (LRU) keys from those that have a TTL set.

Can you please advise which policy should fit best for tempo based on amazon doc?
https://docs.aws.amazon.com/whitepapers/latest/database-caching-strategies-using-redis/evictions.html

[electron0zero]

we don't set TTL, and we use memcached internally so I don't have enough know-how to recommend a config for Redis.

[edgarkz]

if anyone looks for redis config, apparently tempo does have ttl configuration which is by default disabled.
if anyone is looking for valid values are:
Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

Thanks everyone