How to pass auth token headers

[swaroopkundeti]

Hello There,

I have different Prometheus origins and I'm trying to cache them using trickster. Below are the configuration settings of my instance which run fine without any problems. Prometheus origins are accessible by Berar token. I could see the testing works in grafana with data source pointing to trickster with auth token in the header. However, my dashboards are not working as expected when I look at logs I see that Prometheus is responding with 401.

Config settings that works but gives out 401

[frontend]
listen_port = 8480

 origins]
    [origins.south]
    origin_url = 'https://us-south.monitoring.example.com/prometheus'
    origin_type = 'prometheus'
    cache_name = 'default'
    max_object_size_bytes = 5242880
    timeseries_retention_factor = 10240
    timeseries_ttl_secs = 216000
    max_idle_conns = 20
    backfill_tolerance_secs = 150


    # is_default = true

    [origins.north]
    origin_url = 'https://north.monitoring.example.com/prometheus'
    origin_type = 'prometheus'
    cache_name = 'default'
    max_object_size_bytes = 5242880
    timeseries_retention_factor = 10240
    timeseries_ttl_secs = 216000
    max_idle_conns = 20
    backfill_tolerance_secs = 150

[metrics]
listen_port = 8481   # available for scraping at http://:/metrics

[logging]
log_level = 'info'

Logs:

time=2021-11-29T18:02:52.979965245Z app=trickster caller=proxy/engines/deltaproxycache.go:481 level=error event="unexpected upstream response" clientRequestHeaders="unsupported value type" upstreamRequestURL=https://south.monitoring.example.ibm.com/prometheus/api/v1/query_range upstreamRequestMethod=POST upstreamResponseBody="Unauthorized\n" upstreamResponseHeaders="{[Cache-Control:s-maxage=86400],[Connection:keep-alive],[Content-Type:text/plain; charset=utf-8],[Date:Mon, 29 Nov 2021 18:02:52 GMT],[Set-Cookie:INGRESSCOOKIE=1638208973.885.7573.983739; Expires=Tue, 30-Nov-21 18:02:52 GMT; Max-Age=86400; Path=/prometheus/api; Secure; HttpOnly],[Strict-Transport-Security:max-age=15724800; includeSubDomains],[Vary:Accept-Encoding],[X-Content-Type-Options:nosniff]}" statusCode=401 clientRequestURL=https://au-syd.monitoring.cloud.ibm.com/prometheus/api/v1/query_range clientRequestMethod=POST upstreamRequestHeaders="{[Accept:application/json, text/plain, */*],[Accept-Language:en-US,en;q=0.5],[Content-Length:241],[Content-Type:application/x-www-form-urlencoded],[Forwarded:for=127.0.0.1, for=127.0.0.1, for=127.0.0.6;proto=https],[Sec-Fetch-Dest:empty],[Sec-Fetch-Mode:cors],[Sec-Fetch-Site:same-origin],[User-Agent:Grafana/8.2.5],[Via:HTTP/1.1 trickster-84995c98ff-9zmwn],[X-B3-Parentspanid:6dce326106bc1e10],[X-B3-Sampled:0],[X-B3-Spanid:9fe7956b30206917],[X-B3-Traceid:d0d26c7d1cc5a2f46dce326106bc1e10],[X-Dashboard-Id:1],[X-Envoy-Attempt-Count:1],[X-Forwarded-Client-Cert:By=spiffe://cluster.local/ns/opsd-monitoring/sa/default;Hash=25ff70198bd49651d2140948b64beeec5151d0e346ac44ea9594a499e0503f95;Subject=\"\";URI=spiffe://cluster.local/ns/opsd-monitoring/sa/grafana],[X-Grafana-Org-Id:1],[X-Panel-Id:42],[X-Request-Id:3bd6e00b-f67f-4279-85ef-505ff8240c88]}"

I tried the below config but fails to load configuration.

frontend:
  listen_port: 9090

backends:
  south:
    origin_url: 'https://south.monitoring.example.ibm.com/prometheus'
    provider: 'prometheus'
    cache_name: 'default'
    max_object_size_bytes: 5242880
    timeseries_retention_factor: 10240
    timeseries_ttl_secs: 216000
    max_idle_conns: 20
    backfill_tolerance_secs: 150
    headers:
        Authorization: 'Bearer abcdefghijklmnopqrstuvwxyz'
  north:
    origin_url: 'https://north.monitoring.cloud.example.com/prometheus'
    provider: 'prometheus'
    cache_name: 'default'
    max_object_size_bytes: 5242880
    timeseries_retention_factor: 10240
    timeseries_ttl_secs: 216000
    max_idle_conns: 20
    backfill_tolerance_secs: 150
    headers:
       Authorization: 'Bearer abcdefghijklmnopqrstuvwxyz'
metrics:
  listen_port: 8481
  
logging:
  log_level: 'info'

Logs

ERROR: Could not load configuration: no valid origins configured

Trickster version: 1.1.5, buildInfo: 2021-02-04T17:58:04+0000 4595bd6a1ae1165ef497251ad85c646dadc8a925, goVersion: go1.14.14, copyright: © 2018 Comcast Corporation

Is there any a way that I can pass the auth tokens with headers in trickster?