Documentation for shipping server logs/metrics via alloy

[josegonzalez]

Once this is setup, there is an OTLP endpoint for collecting logs/metrics. That said, there isn't really an end-to-end setup for how to configure this with Alloy. Would it be possible for someone to write a short tutorial on how to do that?

[tpaschalis]

Hey there @josegonzalez, thanks for reaching out! 👋

We have a similar tutorial here: https://grafana.com/docs/alloy/latest/collect/opentelemetry-to-lgtm-stack/

Could you take a look and see if it answers your question?

[josegonzalez]

Somewhat? It's not super clear how to grab the actual log data (or even stuff like server metrics) to ship over, or if alloy is even the correct way.

I only point at Alloy as the grafana agent is deprecated in favor of it.

[hero-david]

Hi, I've found the docs helpful, but they seem to tackle different specific hello-world-esque use cases.

I have been reading through the Collect Kubernetes logs documentation, which demonstrates how to do so with Loki components. This is also the recommended way of retrieving logs from infrastructure per the Choose a component page. However, these guides fall flat as only an OTEL collector for GRPC and HTTP is exposed in otel-lgtm container.

I've played around, trying to forward Loki logs to the otel-lgtm docker container, thinking that the HTTP OTEL endpoint might serve /loki/* routes (Ie, so that we can push to http://host:4318/loki/api/v1/push, but no cigar. Hence I ended up here looking for some guidance!

I get the feeling that we may be able to do the following;

1. Expose port 3100 from the container in order to ship logs to Loki (Possibly not recommended?)
2. Convert Loki formatted logs into OTEL format and push those via 4318 (Also possibly not recommended?)

• There is a otelcol.receiver.loki which receives Loki log entries and converts them to the OpenTelemetry format. I managed to ingest OTEL logs using it, and my collector alloy config looks like this (It's missing a few key labels, like the OTEL mandatory service_name:
  

It would be great to know which of these means of ingesting Loki logs is preferred, and perhaps a brief note in the README of this project would go down a treat to help inform people moving forwards! :-)
In all transparency, I'm a bit new to OTEL, so there's a lot of learning going on, just hoping to make that journey easier for others where possible!