What's the current industry standard for Golang logging libraries?

[okdas]

This research has originally been done in #181, but we still want to share this with the community in discussions for future reference.

---

There are so many different logging libraries!

What existing blockchain clients use

go-etherium: copied log15 into the codebase and made some modifications
Erigon: also fork of log15 https://github.com/ledgerwatch/log/tree/master/v3
AVAX: uber-go/zap
Algorand: logrus
CosmosSDK: zerolog

Comparison

rs/zerolog and uber/zap are libraries that avoid allocations and reflection, as a result they are incredibly fast. There is a traid off though in usability when going this route - because in order to avoid allocations one must specify a datatype for each field in logging event structure. uber/zap has a SugaredLogger mode when it still supports printf-like syntax while still being pretty quick.

logrus seems to be the most popular solution, also a pioneer library that was most adopted (probably still is most popular?), with very human-friendly API. As it is imported in many existing software, maintainers avoid API-breaking changes that would be necessary to improve performace. Perf is still decent though!

log15 is an old library that is no longer maintained. Hence it is forked by software that has been around for a while (such as geth). Its maintainer helped with development of github.com/go-kit/log, which is a part of gokit microservice framework.

Final thoughts

I think we need to either use rs/zerolog or uber-go/zap as they are not only most performant, but also being recommended by logrus maintainers. Though we don't necessarily need a very performant library for day-to-day operations, I can see a scenario when we want to dump all network messages client recieves into kafka/clickhouse or something similar for further analysis, and it'd be nice to have a smaller hit on performace.

Please visit following pages to check out the API:

• https://github.com/uber-go/zap#quick-start
• https://github.com/rs/zerolog#getting-started

A few notes I would like to add:

1. Let the loglevel be configurable via argument and/or environment variable, with INFO by default;
2. Similarly, it might be benificial to let the user switch between logging output types - human-readable and JSON. I'd keep JSON by default as this is preferred for logging aggregation tools, and we have loki on local setup to run queries against logs.
3. We should utilize logging structures when it makes sense to provide additional context or make it easier to write queries. For example, when log line is related to particular block, it makes sense to include BlockID or BlockHash as one of the arguments.

[Olshansk]

Our team had an internal discussion about this topic which I wanted to capture here

We've decided to go with zerolog for two reasons:

1. Less cognitive load for those coming from Cosmos
2. Seems to be the "latest and greatest" from general discussions online
3. Syntax between zap and zerlog is very similar

Things we will keep in mind during future discussions and design decisions are:

• Should we create a single telemetry / observability module for everything. If so, what should it be called?
• Keep OpenTelemtry in mind as we keep developing
• Automatically inject metadata (log line, source module, etc...) on every log
• Use structured JSON logging by default
• Make logging be an asynchronous low-priority process so it doesn't interfere with the main process
• Have the ability to dump all the logs into a database like ClickHouse or other

This work is not currently tracked in any Github issues but will be in the future.