Support UTF-8 in metric and label names

[ywwg]

As raised in prometheus/client_ruby#306, we will need some way to encode UTF-8 metric and label names in pushgateway URLs. This may be tricky because the current method does allow for encodings of job names and label values, and any solution would have to be backwards compatible with the current syntax. Thoughtful design work is needed.

See: https://github.com/prometheus/pushgateway#url

Feature request

Use case. Why is this important?

UTF-8 Support is coming to prometheus and we need to cover this usecase.

[fedetorres93]

The path for pushing metrics into the Pushgateway looks like

/metrics/job/<JOB_NAME>{/<LABEL_NAME>/<LABEL_VALUE>}

Currently, job names and label values can be encoded with base64url, in which case job or the label name must be suffixed with @base64. For example, using the grouping key job="example",first_label="foo",second_label="bar":

/metrics/job@base64/ZXhhbXBsZQ==/first_label@base64/Zm9v/second_label@base64/YmFy

Some ideas I considered for encoding UTF-8 label names in URLs, using the same base64url approach:

• Encode the label name using base64url and add a new suffix (for example, @base64name) so that label names are decoded accordingly when the URL is parsed. We can have another suffix for the case where both the label name and the label value are base64url encoded, and keep the original @base64 for when only the value is encoded. For example, using the grouping key job="example",first.label="foo":

/metrics/job/example/Zmlyc3QubGFiZWw=@base64name/foo

• Use a query parameter (like base64names=true) to indicate label names are base64url encoded. This can also coexist with the current @base64 suffix used for label values. Disadvantage: it will be all or nothing.

/metrics/job/example/Zmlyc3QubGFiZWw=/foo?base64names=true

Metric and label names in the request body are validated using functions from the common library that already support UTF-8.

[ywwg]

is there content negotiation with calls to pushgateway, or do people just hit the endpoint? (What if someone sends either of these formats to an older endpoint that doesn't understand the new grammar?)

[fedetorres93]

is there content negotiation with calls to pushgateway, or do people just hit the endpoint? (What if someone sends either of these formats to an older endpoint that doesn't understand the new grammar?)

There's no content negotiation while pushing metrics into the Pushgateway. The URL is parsed as is, splitting the labels with / as the delimiter, then checking if the name contains the @base64 suffix and, once the suffix is trimmed, checking if the label name itself is valid. So if someone sends either of these formats to an older endpoint, the push would be rejected

[beorn7]

So yeah, the base64 trick was introduced prior to 1.0.0. So we never had the situation where a PGW of v1+ would be confronted with a newer version of the push protocol it couldn't understand.

Content negotiation with push is hard, because you need multiple round trips for it.

Ideally, we find a solution that still works for older v1+ PGWs transparently. But I have currently no idea how to do that.

Or we do something that newer pushers have to retry with a legacy version of the names if they get an error back…

[beorn7]

Idea: Let's just use the already proposed bespoke text escaping instead of using base64 escaping. A PGW with UTF-8 support enabled will simply unescape.

Pros:

• It's just another escaping mechanism.
• Better readable than base64 (only special characters appear "weird").
• Backwards compatible (as was the intention when designing this escaping mechanism).
• No new syntax required in the URL.

Cons:

• Now we could have two different escaping mechanisms in the same URL (if a name and a value contains special characters). But this only matters if a human looks at the URL, or a URL has to be hand-crafted.
• In the unlikely case of an existing name that looks like an escaped name (i.e. the original name starts with U__ and happens to have a valid escape sequences), we would falsely unescape.

[beorn7]

To not introduce a (technically) breaking change, we can make the UTF-8 support opt-in for now (finally feature flags in PGW! ;).

[ywwg]

I think it is fine to look for "U__" and unescape automatically if we see it. We decided in the design that it is unlikely people will name metrics like that, and if they do and the unescaping fails, we just return the name as-is anyway

[fedetorres93]

Yeah, I think the escaping alternative makes more sense. +1 on the opt-in. I'll start looking into it, thanks for the feedback!

[beorn7]

Implementation merged. I'll try to cut a new release next week or so.

[beorn7]

Maybe we are not quite there yet. UI needs to know about the new syntax, too. While this is still benign:

How about this:

[ywwg]

This seems like a different, though related issue? Is this a UI in the pushgateway? I'd prefer to leave this bug as closed and open a new one in the right place for this.

[beorn7]

Yes, this is the PGW UI (and the PGW has its own UI, no code pulled in from other repos). I do think this is part of the UTF-8 support. A new bug would also be filed in this repo, so I think keeping this open is easier.

[beorn7]

It should also be pretty easy to fix. Essentially just quoting the name, whenever is needed, and then escape quotes within a name.