Add runtime metrics to prometheus

[vkuznet]

Following approach of https://github.com/arl/statsviz/tree/1ff44fae9a7555de21fe9ad6fa09787373cf23fa we can add runtime metrics to our metrics endpoint and expose them to Prometheus. For that we should get runtime.MemStats object and convert it to our metrics code base web/metrics.go

The list of supported runtime metrics can be found here

[d-ylee]

For this, are we converting the values in runtime.MemStats, then formatting it in accordance to Prometheus metrics? Would these be steps to add this to our metrics endpoint?

1. Add the runtime metrics as fields to the Metrics struct:
   

   dbs2go/web/metrics.go

   Line 96 in bc37f62

   type Metrics struct {

2. Then add the metrics to the struct:
   

   dbs2go/web/metrics.go

   Line 130 in bc37f62

   func metrics() Metrics {

3. Then format the metrics so that Prometheus would be happy parsing them?
   

   dbs2go/web/metrics.go

   Line 204 in bc37f62

   func promMetrics(prefix string) string {

[vkuznet]

yes, it is correct procedure.

[d-ylee]

@vkuznet I tried this method using Prometheus' Go Client. I implemented this temporarily on the /runtimemetrics endpoint Would this be something we want to use?
d-ylee@0619377

[vkuznet]

Dennis, the new code looks solid to me, and since you implemented new metrics on dedicated end-point we can proceed with their deployment and build new dashboard for them. Then, we can compare existing dashboard with new one, and, finally, merge them together. Feel free to merge and proceed with deployment on testbed. But since you put them on dedicated end-point we'll need an additional exporter to scrape them for Prometheus since k8s manifest file only allow single end-point/port to use via annotations, see https://github.com/dmwm/CMSKubernetes/blob/master/kubernetes/cmsweb/services/dbs2go-global-r.yaml#L60-L62

[d-ylee]

@vkuznet Would it be an issue if the metrics scraped have the same name as the original metric? Or should I modify the prefix on the new endpoint?

[vkuznet]

Even though Promeethes us very flexible I think it would be wise to setup originally different prefix. In a long run you want to merge both metrics under single end-point and having single prefix but until everything is tested I do not want to break existing (production) dashboards, e.g. https://monit-grafana.cern.ch/d/kJOvR2h7k/dbs2go?orgId=11