network/discovery: rate limit skipped peers log #1803
Conversation
network/discovery/metrics.go
Outdated
| metricSkippedPeers = promauto.NewCounterVec(prometheus.CounterOpts{ | ||
| Name: "ssv:network:discovery:skipped_peers", | ||
| Help: "Counts skipped peers", | ||
| }, []string{"enr", "peer_id", "err"}) |
There was a problem hiding this comment.
Having peer ID or an ENR as a label is problematic because of its high cardinality (Prometheus struggles with this). We could use something such as ssv:network:discovery:skipped_peers_total for now if a total count would help. If not, we need some other solution.
There was a problem hiding this comment.
@zktaiga reverted the change and added the log rate limit
nkryuchkov
changed the title
|
this looks good, but the map will never clean and may grow infinitely instead of implementing cleanup, i suggest just using https://github.com/jellydator/ttlcache instead (because we already import it) |
There was a problem hiding this comment.
clean the cache, maybe use some existing ttl cache like @moshe-blox suggested
|
@y0sher @moshe-blox I heard from @zktaiga we decided not to proceed with this solution, so can I close this? |
@zktaiga reported that this log makes up to 40% of stage logs