Add support for multiple formulas in DD metricprovider

[y-rabie]

Summary

Since the DD API already supports multiple formulas, the changes are supported. Namely, they are:

• There needs to be added a field formulas in addition to the existing formula, where they are mutually exclusive (same pattern as with query and queries).

• When formulas are used, the result will be a vector instead of a scalar value, similar to Prometheus' case. But where in Prometheus' case only result[0] made sense to access, all elements of result should be accessible in the condition strings.

Nothing else needs to be changed.

This can be, and in-fact is, backward-compatible, but may require a new CRD version. I do not have full grasp when it comes to the versioning part, but I've made a PR with all the required changes and new unit test. Check there for further details.

Use Cases

I think conditions right now are limited in accessing the result of a single formula. I might need to assert multiple conditions, regarding the same metric with the same set of queries. I don't know why that shouldn't be available, in the same vein as you would write a normal Datadog Monitor.

A fairly naive example would be: checking the rate of 2xx requests to total requests (2xx and 4xx for example), while asserting that the total volume of request is above a certain absolute threshold.

The two conditions need to be coupled in the same metric, and this cannot be done with the current setup, so that if one condition fails, the whole metric fails.

Generally, however, this would allow for more complex conditions.

It would look like the following:

    # result[0] is the rate, result[1] is the absolute total volume
    # Asserting the success rate is greater than 0.90 WHILE the total volume of requests is  more than 500
    failureCondition: default(result[0], 1) < 0.90 && default(result[1], 0) >= 500

    failureLimit: 3
    provider:
      datadog:
        apiVersion: v2
        aggregator: last
        interval: 1m
        queries:
          a: |
            sum:haproxy.frontend.http.responses.count{code:2xx}.as_count()

          b: |
            sum:haproxy.frontend.http.responses.count{code:4xx}.as_count()

        formulas:
          - "a/(a+b)"
          - "a+b"

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.