Proposal to change guardian logging from Google Cloud Logs and metrics storage

[SEJeff]

Guardian Logging and Metrics Change Proposal

Currently, guardians can opt in to send their guardian logs centrally via the following flags:

• --telemetryKey
• --telemetryServiceAccountFile
• --telemetryProject

This sends all guardian logs to google cloud logs for usage by any guardian in the protocol for troubleshooting and improving the network. The protocol generates approximately 30T of logs monthly and would like to keep them for at least 30 days.

Additionally, there isn't a central location for guardian metrics. Each guardian maintains their own metrics. It is often useful for wormhole contributors to view metrics for a mainnet guardian to ensure things are running correctly network-wide.

Problem

1. The log search is slow and cumbersome to navigate.
2. There isn't a way to view metrics without running a guardian
3. Onboarding new users from various wormhole contributors is difficult.
4. Google Cloud logs is quite expensive for the current volume and desired retention.
5. Dashboards and metrics for the guardians sometimes need to be available to contributors not necessarily running their own guardian.

Solution

The Wormhole Foundation uses Grafana Enterprise and sends logs to Loki. The metrics could (in theory) be sent to a prometheus push gateway or guardians could use the remote write feature for grafana enterprise metrics.

Benefits:

1. Loki search is label, text, and time based. As a result, Loki is fast since it expects structured logs. The wormhole guardian logs are structured using zap so this should work without any modifications.
2. The Foundation will have a Grafana enterprise instance that can present metrics and read-only dashboards to authorized parties.
3. Grafana supports federated SSO. This allows trusting the authoritative SSO for each set of contributors. Onboarding a new user from an existing contributor happens within the org specific SSO solution instead of manual work for the foundation or contributors. Onboarding a new organization is just adding their SSO endpoint and configuring grafana to use it.
4. Grafana gives you more for a lower price. I asked for a quick quote for 30T ingested monthly with a 30 day retention and the price was ~30% of the cost of the current Google Cloud Logs bill alone (with no optimization of the log volume).
5. Grafana is included as part of Grafana Enterprise, so the foundation will have a central graph / dashboard instance.

Deliverables

1. Ensuring the best (consistent) set of labels are used on logs to make indexing and filtering work really well (e.g.: adding a security label to all security relevant logs). The goal for this deliverable is consistency in the labels.
   1.1 Ensuring the various watchers use similar labels and logs
2. Identifying useless logs to remove, logs that are spammy, or debug logs with a non-debug loglevel and fix them to reduce log spam. The goal of this deliverable is less overall log volume.
3. Adding support for the guardian node to send metrics to loki. This can be done inline as a logger using the promtail loki client.
   3.1. Get guardians who opt-in to sending telemetry to migrate to loki
   3.2 Remove gcp log support from the guardian node

[dancamarg0]

A few comments from my team member:

I would try to set a standard for all operators regarding:

• Tags - Because if cardinality is high we can lose performance, and it will also make filtering more complex
• Consider using other sender options even tho they may recommend promtail (it's doable to implement promtail), but we already have fluentd setup, and others may be using a different option, like fluentbit

The parsing of the logs should be defined by them and not be let up to whoever implements it

[SEJeff]

@dancamarg0 the promtail integration is at the code level, not the operator level. This is using the loki specific promtail library, not the promtail executable.

Currently, the google cloud logs bits is also at the code level. If you use fluentbit (great software btw!) but enable telemetry, it is entirely indepenent and sends the logs from the guardiand process directly to google cloud logs. It also logs to stderr and that is what fluentbit would use.

This is not requiring the node operators to do anything other than configure a flag such as --telemetryKey or --telemetryServiceAccountFile

Regarding the labels and messages, I was referring to the ones in the code, which are different between different watchers, which is confusing and annoying. Compare these NEAR metrics vs these evm metrics vs these cosmwasm metrics. Just want to bring some semblance of sanity and uniformity between them all.

[daviesalex]

+1 to this proposal. Better, cheaper, and more scalable for multiple contributors to have equal access during development and incident response. Loki is a great UI long term.

One thing to consider is if we need all the current roughly 30TB logs or not in a system like Loki, and if we want to store them for more than 30 days. A large chunk of that 30TB was added in the last year to aid security incident response, which is one time queries that generally involve intensive analysis on a subset of logs 'offline' (vs queries in a web UI like Loki). This probably mostly boils down to cost for the Foundation, but it would be possible to imagine a way to route all logs to a much more scalable / cheaper Kafka -> GCS/S3 archive and then just the subset we want to show in graphs to Loki. I think you could make that decision in the Kafka layer so changes to it would not require Guardian code changes.

[SEJeff]

One thing to consider is if we need all the current roughly 30TB logs or not in a system like Loki

My suggestion is that we follow your suggestion, but not all at once. What do you think of something like this to get measurable improvements out quickly?

1. Like for like move from google cloud logs to something equivalent or better (like Loki). Fast, relatively easy, and gives us more leeway.
2. Work on lowering volume of logs. This is made much easier with some of the framework @tbjump has done in his great node refactor stack of PRs. This can in theory happen in parallel with step 1 by a different engineer. @tbjump and I planned on doing this once more of his stack is merged.
3. Analyze what logs need to be searchable, and which don't (this is more up to the active contributors and devs and is quite subjective is what I fear).
   3.1 We figure out the business side (redpanda or not, etc) and technical implementation side. This likely will entail some relatively complex logic in the guardian which isn't great, or some sort of a proxy or something that's a frontend for all logs that then splits out some to a streaming service and some to loki.

Step three requires a bit more thought and would slow down the execution of steps one and two. I propose we do steps one and two as soon as feasible. Then we try to get step three done before contract renewal (the quote from grafana only lets you pay for enterprise annually. I asked and was told no for monthly payments). That said, it is a very good idea and I'm for anything that lessens the burden on the foundation.

Does that work @daviesalex?

[ab-everstake]

We are enthusiastic about the concept of log aggregation and analysis; however, it is crucial to acknowledge that logs may contain sensitive information, eventually.

In our infrastructure, we leverage the combined capabilities of Gcloud and Loki for various tasks. Gcloud provides typical benefits of cloud service ( faster provision, no need to setup/maintain ), while Loki in self-hosted mode offers compelling advantages such as really low costs.

To ensure utmost flexibility for everyone, we vote for migrating to Loki for cost reduction.

[SEJeff]

Thankyou for your input @ab-everstake I suspect the idea medium to long term is to self host this stack, but the foundation is not yet ready to do that.