In this tutorial, we will walk through how to get started with instrumenting the emojivoto application, a well known microservice example running locally on a Kubernetes cluster.
The following tools are required to run this tutorial:
- Kind to run a local Kubernetes cluster with Docker container nodes.
- Kubernetes CLI (kubectl) to run commands against Kubernetes clusters.
Create a new local Kubernetes cluster, by running the following command:
kind create cluster
The Kubernetes cluster will run the emojivoto applications and a jaeger UI for visualizing the OpenTelemetry traces:
The different emojivoto applications are communicating via gRPC. Instrumenting those applications would allow us to view the gRPC requests between the applications.
Run the following command:
kubectl apply -k github.com/keyval-dev/opentelemetry-go-instrumentation/docs/getting-started/emojivoto
Install Jaeger UI by running:
kubectl apply -f https://raw.githubusercontent.com/keyval-dev/opentelemetry-go-instrumentation/master/docs/getting-started/jaeger.yaml -n emojivoto
This command installs Jaeger as a new Deployment and an additional Service that we will use later for accessing the Jaeger UI.
In a real world application, you would probably want to send the tracing data to OpenTelemetry collector instead of directly to Jaeger.
Apply the automatic instrumentation to the emoji
, voting
, and web
applications by executing the following command:
kubectl apply -f https://raw.githubusercontent.com/keyval-dev/opentelemetry-go-instrumentation/master/docs/getting-started/emojivoto-instrumented.yaml -n emojivoto
Now all that left to do is to perform some actions on the target application that will cause the creation of detailed distributed traces.
Port forward to the frontend service:
kubectl port-forward svc/web-svc 8080:80 -n emojivoto
Go to http://localhost:8080
, and click the view the leaderboard button.
First, port forward to the Jaeger UI:
kubectl port-forward svc/jaeger 16686:16686 -n emojivoto
Then, open the Jaeger UI in your browser by navigating to http://localhost:16686/
After selecting the emojivoto-web
service, you should see the following traces:
Let's start with a simple trace, click on one of the /api/vote
traces, you should see something like the following:
A few things worth noticing in this trace:
- In process context-propagation: We can see that the
web
application got an HTTP request (the root span) which caused two sequential gRPC requests to theemoji
andvoting
services. - Cross process context-propagation: The automatic instrumentation adds the relevant headers to gRPC / HTTP requests in order to make traces distributed across processes. In Jaeger, different applications are marked in different colors.
- Extremely low overhead: The entire trace took 3.1 milliseconds to complete. This show that using eBPF for instrumentation adds extremely low overhead.
- Following OpenTelemetry Specifications: You can click on any span in the trace to view the different attributes. All the spans produced by the automatic instrumentation follows the OpenTelemetry specification and therefore will work with any OpenTelemetry compatible backend.
Now we will view a more complex trace, search for traces for the /api/leaderboard
endpoint.
You can quickly find them by clicking on the points with the highest duration in Jaeger's main screen.
We can get a pretty good understanding of how the leaderboard feature works by looking at this trace:
- First, the web service will perform a gRPC request to the
voting
service to get a list of the available emojis. - Second, The web service will loop over the received list of emojis and for every item on the list it will perform a gRPC request to the
emoji
service to get the amount of votes for the current item. - As you can see this happens sequentially, which is one of the reason the leaderboard endpoint takes about 150ms to complete.
Notice that we did not change any application code to get those traces, we are using the exact same containers from the emojivoto project.
The easiest way to apply this automatic instrumentation for any application is by using a control plane such as Odigos.
For more details visit the Odigos website.
Delete the Kubernetes cluster:
kind delete cluster