[SOLVED] Howto to run HotROD with a custom Jaeger instance created from jaeger-operator helm + Jaeger CRD #4725

tringuyen-yw · 2023-09-05T04:50:36Z

tringuyen-yw
Sep 5, 2023

Hi,

Installed jaeger-operator, helm chart. Then I created an AllInOne instance of Jaeger

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: monojaege
  namespace: jaeger
spec:
  strategy: allInOne
  allInOne:
    image: jaegertracing/all-in-one:latest
    options:
      log-level: debug
  storage:
    type: memory # valid values: memory, Cassandra, Elasticsearch, Kafka
    options:
      memory:
        max-traces: 100000
  ingress:
    enabled: false

I would like to run the HotROD app against this Jaeger instance. By generating the YAML of the example-hotrod from the kustomize package . The example-hotrod itself, unaltered, works perfectly.

However, I would like to connect HotROD to my own Jaeger instance. I edited the kuztomize generated YAML to remove the resources related to the Jager instance. I have tried to add some environment variables to connect my on Jaeger instance. The HotROD deployment looks like this:

apiVersion: v1
kind: Namespace
metadata:
  name: jaeger-demo-hotrod
---
apiVersion: v1
kind: Service
metadata:
  name: hotrod-service
  namespace: jaeger-demo-hotrod
spec:
  ports:
    - name: frontend
      port: 8080
      protocol: TCP
      targetPort: frontend
  selector:
    app: hotrod-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hotrod-app
  name: hotrod-app
  namespace: jaeger-demo-hotrod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hotrod-app
  template:
    metadata:
      labels:
        app: hotrod-app
    spec:
      containers:
        - name: hotrod-app
          image: jaegertracing/example-hotrod:latest
          imagePullPolicy: IfNotPresent
          args:
            - all
            - '--verbose'
            - '--jaeger-ui=http://monojaeger-query.jaeger.svc:16686'
          env:
            # Jaeger client env vars:
            # https://github.com/jaegertracing/jaeger/blob/main/internal/jaegerclientenv2otel/envvars.go
            - name: OTEL_EXPORTER_JAEGER_AGENT_HOST
              value: localhost
              #value: monojaeger-agent.jaeger
            - name: OTEL_EXPORTER_JAEGER_AGENT_PORT
              value: "6831"
            - name: OTEL_EXPORTER_JAEGER_ENDPOINT
              value: monojaeger-collector-headless.jaeger.svc:14250
            - name: OTEL_EXPORTER_OTLP_INSECURE
              value: 'false' # false: user https, true:
          ports:
            - containerPort: 8080
              name: frontend
            - containerPort: 8081
              name: customer
            - containerPort: 8083
              name: route
          resources:
            limits:
              cpu: 100m
              memory: 100M
            requests:
              cpu: 100m
              memory: 100M

The HotROD app is not able to connect to my Jaeger instance. It seems it always POST trace data to localhost. I have also tried to add a jaeger-agent sidecar in the same pod as the HotROD container. Same behavior. The HotROD pod logs looks like this:

2023-09-04T16:14:23.789Z	INFO	[email protected]/command.go:970	Using expvar as metrics backend
2023-09-04T16:14:23.789Z	INFO	[email protected]/command.go:940	Starting all services
2023-09-04T16:14:23.790Z	INFO	customer/server.go:56	Starting	{"service": "customer", "address": "http://0.0.0.0:8081"}
2023-09-04T16:14:23.790Z	INFO	route/server.go:55	Starting	{"service": "route", "address": "http://0.0.0.0:8083"}
2023-09-04T16:14:23.790Z	INFO	frontend/server.go:77	Starting	{"service": "frontend", "address": "http://0.0.0.0:8080"}
2023-09-04T16:14:23.790Z	INFO	driver/server.go:63	Starting	{"service": "driver", "address": "0.0.0.0:8082", "type": "gRPC"}
2023/09/04 16:15:02 traces export: Post "http://localhost:4318/v1/traces": dial tcp [::1]:4318: connect: connection refused

QUESTION: is it possible to configure HotROD to connect to a custom Jaeger instance?

Among the custom parameters, I have better success with the hotrog argument --otel-exporter=stdout. The trace data does appear in the HotROD pod logs. But obviously, nothing could be analyzed using the Jaeger UI.

Answered by yurishkuro

Sep 5, 2023

It works without the path if you use higher-level variable OTEL_EXPORTER_OTLP_ENDPOINT

jaeger/examples/hotrod/docker-compose.yml

Line 23 in f8c1523

- OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318

View full answer

yurishkuro · 2023-09-05T05:06:56Z

yurishkuro
Sep 5, 2023
Maintainer

QUESTION: is it possible to configure HotROD to connect to a custom Jaeger instance?

standard OTEL SDK env vars should be working https://github.com/open-telemetry/opentelemetry-go/tree/02616a25c68e674a04df01a1b7ba185a2db8cd5a/exporters/otlp/otlptrace#environment-variables

2 replies

tringuyen-yw Sep 5, 2023
Author

standard OTEL SDK env vars should be working https://github.com/open-telemetry/opentelemetry-go/tree/02616a25c68e674a04df01a1b7ba185a2db8cd5a/exporters/otlp/otlptrace#environment-variables

Thank you. I had playaround many times with OTEL_EXPORTER_JAEGER_ENDPOINT at one point I even set a bogus endpoint. Hoping to see an error message in the HotROD pod logs. But there was none. The link you gave
shows another variable OTEL_EXPORTER_OTLP_TRACES_ENDPOINT maybe this one would work.

Just to make 100% sure, can you please suggest what a good value for endpoint should look like? http, with which port number? Should UDP be used? Any recommended URI scheme? Which OTEL environment variables should I customize?

yurishkuro Sep 5, 2023
Maintainer

The latest HotROD does not support Jaeger exporter, only OTLP exporter (by default over HTTP). So you will want to use OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. Default is http://localhost:4318

tringuyen-yw · 2023-09-05T19:59:45Z

tringuyen-yw
Sep 5, 2023
Author

@yurishkuro Goes a bit better but still fail. set the envvar for HotROD app Deployment as:

          env:
            # OpenTelemetry-Go OTLP Span Exporter env vars:
            # https://github.com/open-telemetry/opentelemetry-go/tree/main/exporters/otlp/otlptrace
            - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
              value: https://monojaeger-collector-headless.jaeger.svc.cluster.local:4318

The HotROD pod logs now acknowledges the new URL but failed by 404

2023/09/05 19:54:14 traces export: failed to send to http://monojaeger-collector.jaeger.svc.cluster.local:4318/: 404 Not Found

Shelling into a pod of the same cluster, netcat shows that the endpoint is working:

nc -zv monojaeger-collector.jaeger.svc.cluster.local 4318

# console output
Connection to monojaeger-collector.jaeger.svc.cluster.local 4318 port [tcp/*] succeeded!

0 replies

tringuyen-yw · 2023-09-05T20:08:14Z

tringuyen-yw
Sep 5, 2023
Author

Working now! OpenTelemetry, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT mentions:

Must end with v1/traces if using OTLP/HTTP

So the correct envvar value for the HotROD deployment template should be:

          env:
            # OpenTelemetry-Go OTLP Span Exporter env vars:
            # https://github.com/open-telemetry/opentelemetry-go/tree/main/exporters/otlp/otlptrace
            # https://opentelemetry.io/docs/concepts/sdk-configuration/otlp-exporter-configuration/#otel_exporter_otlp_traces_endpoint
            - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
              value: https://monojaeger-collector-headless.jaeger.svc.cluster.local:4318/v1/traces

0 replies

yurishkuro · 2023-09-05T20:31:10Z

yurishkuro
Sep 5, 2023
Maintainer

It works without the path if you use higher-level variable OTEL_EXPORTER_OTLP_ENDPOINT

jaeger/examples/hotrod/docker-compose.yml

Line 23 in f8c1523

- OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318

0 replies

tringuyen-yw · 2023-09-05T22:36:41Z

tringuyen-yw
Sep 5, 2023
Author

Thanks, OTEL_EXPORTER_OTLP_ENDPOINT is slightly better (no need to define /v1/traces in the endpoint path). But more fragile!

          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: https://monojaeger-collector-headless.jaeger.svc.cluster.local:4318

The HotROD pod logs shows

2023/09/05 22:16:40 traces export: Post "https://monojaeger-collector-headless.jaeger.svc.cluster.local:4318/v1/traces": http: server gave HTTP response to HTTPS client

As a result, the HotROD app failed to push any trace data to the jaeger collector. Had to change https to http. Strangely enough, when I was using OTEL_EXPORTER_OTLP_TRACES_ENDPOINT with the full URL https://monojaeger-collector-headless.jaeger.svc.cluster.local:4318/v1/traces. There was no error in log warning about https/http mismatch. But somehow the HotROD client was able to upload trace data to jaeger collector.

Could that be the OpenTelemetry SDK figured out the jaeger collect doesn't support https and transparently convert it's POST request to http?

0 replies

thatmlopsguy · 2024-08-03T17:35:06Z

thatmlopsguy
Aug 3, 2024

@tringuyen-yw can you share the final hotrod deployment with the all required env variables. Thanks.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jaeger - Distributed Tracing Platform

[SOLVED] Howto to run HotROD with a custom Jaeger instance created from jaeger-operator helm + Jaeger CRD #4725

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Jaeger - Distributed Tracing Platform

[SOLVED] Howto to run HotROD with a custom Jaeger instance created from jaeger-operator helm + Jaeger CRD #4725

tringuyen-yw Sep 5, 2023

Replies: 6 comments · 2 replies

yurishkuro Sep 5, 2023 Maintainer

tringuyen-yw Sep 5, 2023 Author

yurishkuro Sep 5, 2023 Maintainer

tringuyen-yw Sep 5, 2023 Author

tringuyen-yw Sep 5, 2023 Author

yurishkuro Sep 5, 2023 Maintainer

tringuyen-yw Sep 5, 2023 Author

thatmlopsguy Aug 3, 2024

tringuyen-yw
Sep 5, 2023

Replies: 6 comments 2 replies

yurishkuro
Sep 5, 2023
Maintainer

tringuyen-yw Sep 5, 2023
Author

yurishkuro Sep 5, 2023
Maintainer

tringuyen-yw
Sep 5, 2023
Author

tringuyen-yw
Sep 5, 2023
Author

yurishkuro
Sep 5, 2023
Maintainer

tringuyen-yw
Sep 5, 2023
Author

thatmlopsguy
Aug 3, 2024