-
Notifications
You must be signed in to change notification settings - Fork 109
/
distributed-tracing.html.md.erb
110 lines (81 loc) · 5.13 KB
/
distributed-tracing.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
---
title: Distributed tracing
owner: Diego
---
Cloud Foundry provides support for distributed tracing, enabling transactions
to be monitored and analyzed across multiple Cloud Foundry components. For each
transaction initiated in the GoRouter, a new trace context is generated and
then propagated to all participating systems.
Each participating system adds a unique identifier to logs, allowing
you to trace logs and debug errors specific to that transaction. Distributed tracing is particularly beneficial when used with log
aggregation tools like Splunk.
## <a id='unique-trace-id'></a> Unique trace identification
For every request in Cloud Foundry, GoRouter generates a unique
`X-Vcap-Request-Id` header. `X-Vcap-Request-Id` is a v4 UUID according to [RFC 4122](http://www.ietf.org/rfc/rfc4122.txt).
If Zipkin is enabled, see [Enabling Zipkin tracing in Cloud Foundry](/adminguide/zipkin_tracing.html).
GoRouter generates B3 headers
for each request unless they were provided by the client:
* `X-B3-Traceid` - 128-bit in length and indicates the overall ID of the trace.
Every span in a trace shares this ID. The `X-B3-TraceId` header is encoded as
32 lower-hex characters.
* `X-B3-Spanid` - 64-bit in length and indicates the position of the current
operation in the trace tree. The `X-B3-SpanId` header is encoded as 16
lower-hex characters.
* `B3` - a combination of `X-B3-Traceid` and `X-B3-Spanid` headers in the
format: `{TraceId}-{SpanId}`
If W3C is enabled through the `router.tracing.enable_w3c` deployment manifest property,
GoRouter generates W3C headers for each request unless they were provided
by the client. W3C headers are generated according to the [W3C headers format](https://www.w3.org/TR/trace-context/):
* `traceparent` - consists of 4 fields in the format:
`{version-format}-{trace-id}-{parent-id}-{trace-flags}`
* `tracestate` - is generated from `router.tracing.w3c_tenant_id` deployment
property and `parent-id` used in the transparent header:
`{w3c_tenant_id}={parent-id}`
If B3 headers are provided and they are in a proper v4 UUID format, GoRouter
uses them for `X-Vcap-Request-Id` and W3C headers. If they are not
provided, GoRouter generates v4 UUID and use it for B3 headers,
`X-Vcap-Request-Id` and W3C headers.
## <a id='distributed-tracing-example'></a> Using trace identification to analyze a specific Cloud Foundry request
In the following example, a request is made to Cloud Foundry to scale an application. This request is handled by several Cloud Foundry components:
1. The request is initially received by GoRouter. GoRouter generates unique trace headers for this request. It then forwards this request with the trace headers to the Cloud Controller.
1. Cloud Controller updates the LRP (Long Running Process) for the application, reflecting the new number of instances. It subsequently sends the LRP request with the appended trace headers to BBS.
1. BBS generates a new request with the trace headers to Auctioneer to schedule a new application instance.
1. Auctioneer selects a Diego Cell that has enough resources to run the application instance. It then sends a request with the trace headers to Rep on that Diego Cell, instructing it to run this new application instance.
1. Finally, Rep generates a request to Garden. This request with the same trace information, instructs Garden to create a container where application processes run.
Each component in this transaction:
* Retains the original request's `trace-id` and forwards it in all downstream requests.
* Generates a unique span-id for each request, which remains exclusive to that particular component.
* Both the `trace-id` and `span-id` are recorded in logs related to that request handling.
This transaction is characterized by a single trace-id that identifies it
across all hops, while each individual hop is marked by its own unique span-id.
The `CF_TRACE` environment variable is used to get the generated B3 trace headers.
```bash
CF_TRACE=true cf scale myapp -i 2
```
The output contains a sequence of requests and responses from Cloud
Foundry. Initially, the CLI retrieves the application GUID, followed by an API
request to scale the application. The response to this request includes B3
headers, which are generated by the GoRouter.
```bash
Scaling app myapp in org myorg / space myspace as admin...
REQUEST:
POST /v3/apps/8e0f1501-ba1f-4efe-af90-b61f2e5a1384/processes/web/actions/scale
...
{
"instances": 2
}
RESPONSE:
HTTP/1.1 202 Accepted
...
X-B3-Spanid: 67bde5d7dcd81f84
X-B3-Traceid: 0c2cc9583f004a4167bde5d7dcd81f84
...
```
You can tell the GoRouter to use user-provided B3 headers. For example:
```bash
cf curl -H "X-B3-Traceid: 8c436ec437e545227fe76a3eba99ca21" -H "X-B3-Spanid: 7fe76a3eba99ca21" -d '{"instances" : 3}' /v3/apps/8e0f1501-ba1f-4efe-af90-b61f2e5a1384/processes/web/actions/scale
```
You must set both X-B3-Traceid and X-B3-Spanid.
The `X-B3-Traceid` can be used to filter logs in aggregation tools to see how this request was handled by the GoRouter, Cloud Controller, BBS, Rep, Auctioneer and how a container for a new application instance was created in Garden.
