Why my metrics_generator not remote writing my metrics of traces?

[evan361425]

I am using distributed service in K8s:

kubectl create namespace test
helm -n test install prometheus prometheus-community/prometheus -f prometheus.yaml
helm -n test install tempo grafana/tempo-distributed -f tempo.yaml

prometheus.yaml:

server:
  extraArgs:
    web.enable-remote-write-receiver: null
alertmanager:
  enabled: false
kube-state-metrics:
  enabled: false
prometheus-node-exporter:
  enabled: false
prometheus-pushgateway:
  enabled: false

tempo.yaml:

storage:
  trace:
    backend: s3
    s3:
      access_key: 'grafana-tempo'
      secret_key: 'supersecret'
      bucket: 'tempo-traces'
      endpoint: 'tempo-minio:9000'
      insecure: true
#MinIO storage configuration
minio:
  enabled: true
  mode: standalone
  rootUser: grafana-tempo
  rootPassword: supersecret
  buckets:
  # Default Tempo storage bucket
  - name: tempo-traces
    policy: none
    purge: false
traces:
  otlp:
    grpc:
      enabled: true
    http:
      enabled: true
  zipkin:
    enabled: false
  jaeger:
    thriftHttp:
      enabled: false
  opencensus:
    enabled: false
# metrics-generator for service maps
metricsGenerator:
  enabled: true
  config:
    registry:
      collection_interval: 15s
      external_labels: {}
      stale_duration: 15m
    processor:
      service_graphs:
        histogram_buckets: [0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 6.4, 12.8]
        max_items: 10000
        wait: 10s
        workers: 10
      span_metrics:
        histogram_buckets: [0.002, 0.004, 0.008, 0.016, 0.032, 0.064, 0.128, 0.256, 0.512, 1.02, 2.05, 4.10]
    storage:
      path: /var/tempo/wal
      wal:
      remote_write_flush_deadline: 1m
      remote_write:
      - url: http://prometheus-server.test.svc.cluster.local/api/v1/write

After using k6 to generate tracing data, I can't find out any metrics for tracing:

docker run --network host --rm -e ENDPOINT=localhost:4317 ghcr.io/grafana/xk6-client-tracing:v0.0.5

What do I miss?

[evan361425]

Here is metrics-generator's logs:

{"caller":"main.go:225","level":"info","msg":"initialising OpenTracing tracer","ts":"2024-06-19T07:10:46.136767326Z"}
{"caller":"main.go:118","level":"info","msg":"Starting Tempo","ts":"2024-06-19T07:10:46.148879664Z","version":"(version=2.5.0, branch=HEAD, revision=46dad3411)"}
{"caller":"server.go:240","grpc":":9095","http":":3100","level":"info","msg":"server listening on addresses","ts":"2024-06-19T07:10:46.150005448Z"}
{"caller":"memberlist_client.go:435","cluster_label":"","level":"info","msg":"Using memberlist cluster label and node name","node":"tempo-metrics-generator-d9ffd6cb8-zj8xv-b5340ce4","ts":"2024-06-19T07:10:46.150692444Z"}
{"caller":"module_service.go:82","level":"info","module":"internal-server","msg":"starting","ts":"2024-06-19T07:10:46.150972109Z"}
{"caller":"module_service.go:82","level":"info","module":"server","msg":"starting","ts":"2024-06-19T07:10:46.151033525Z"}
{"caller":"module_service.go:82","level":"info","module":"memberlist-kv","msg":"starting","ts":"2024-06-19T07:10:46.151389731Z"}
{"caller":"module_service.go:82","level":"info","module":"overrides","msg":"starting","ts":"2024-06-19T07:10:46.151433439Z"}
{"caller":"module_service.go:82","level":"info","module":"metrics-generator","msg":"starting","ts":"2024-06-19T07:10:46.151619729Z"}
{"caller":"memberlist_client.go:541","level":"info","msg":"memberlist fast-join starting","nodes_found":6,"to_join":2,"ts":"2024-06-19T07:10:46.151924811Z"}
{"caller":"memberlist_client.go:561","elapsed_time":"1.581865ms","joined_nodes":2,"level":"info","msg":"memberlist fast-join finished","ts":"2024-06-19T07:10:46.15288343Z"}
{"caller":"memberlist_client.go:573","join_members":"dns+tempo-gossip-ring:7946","level":"info","msg":"joining memberlist cluster","phase":"startup","ts":"2024-06-19T07:10:46.152917471Z"}
{"caller":"basic_lifecycler.go:297","instance":"tempo-metrics-generator-d9ffd6cb8-zj8xv","level":"info","msg":"instance not found in the ring","ring":"metrics-generator","ts":"2024-06-19T07:10:46.152983971Z"}
{"caller":"app.go:214","level":"info","msg":"Tempo started","ts":"2024-06-19T07:10:46.153247844Z"}
{"caller":"memberlist_client.go:580","elapsed_time":"3.731267ms","level":"info","msg":"joining memberlist cluster succeeded","phase":"startup","reached_nodes":6,"ts":"2024-06-19T07:10:46.156655072Z"}

[evan361425]

This is my memberlist output:

the value of tempo_distributor_metrics_generator_clients:

[joe-elliott]

Can you go through the steps of our troubleshooting doc:

https://grafana.com/docs/tempo/latest/troubleshooting/metrics-generator/

[evan361425]

I have fixed the issue:

• Tempo required setting overrides to enable metrics-generator
  https://github.com/grafana/tempo/blob/main/modules/distributor/distributor.go#L365-L367
• Tempo's helm values default using file to setup override:
  https://github.com/grafana/helm-charts/blob/tempo-distributed-1.11.0/charts/tempo-distributed/values.yaml#L1359-L1360

After setting up the overrides as partial documents mentioned, it will not be enabled:

# WRONG!
global_overrides:
  per_tenant_override_config: /runtime-config/overrides.yaml
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]

You should remove per_tenant_override_config.