Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High cardinality from metric "rest_client_request_latency_seconds_sum" #269

Open
shaoxt opened this issue Jun 25, 2021 · 1 comment
Open

High cardinality from metric "rest_client_request_latency_seconds_sum" #269

shaoxt opened this issue Jun 25, 2021 · 1 comment

Comments

@shaoxt
Copy link
Collaborator

shaoxt commented Jun 25, 2021

Is this a BUG REPORT or FEATURE REQUEST?:
BUG

What happened:



rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D", verb="GET"} | 0.026166687
-- | --
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/eviction", verb="POST"} | 3.3592428640000005
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1/nodes", verb="GET"} | 15600.239082934992
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1/nodes/%7Bname%7D", verb="PATCH"} | 1.2786773510000002
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1/pods?fieldSelector=%7Bvalue%7D", verb="GET"} | 130.45066546799998
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1?timeout=32s", verb="GET"} | 0.009988171000000002
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api?timeout=32s", verb="GET"} | 3.033722622
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/activemonitor.keikoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.002481853
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/addonmgr.keikoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.599171469
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/admiral.io/v1alpha1?timeout=32s", verb="GET"} | 0.002490315
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/admissionregistration.k8s.io/v1?timeout=32s", verb="GET"} | 0.002397739
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002413015
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apiextensions.k8s.io/v1?timeout=32s", verb="GET"} | 1.098874921
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002451179
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apiregistration.k8s.io/v1?timeout=32s", verb="GET"} | 0.001346273
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apiregistration.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002670913
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apps/v1/namespaces/%7Bnamespace%7D/daemonsets/%7Bname%7D", verb="GET"} | 61.567477207999985
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apps/v1?timeout=32s", verb="GET"} | 1.298807949
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/argoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.499365619
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/authentication.istio.io/v1alpha1?timeout=32s", verb="GET"} | 0.399400264
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/authentication.k8s.io/v1?timeout=32s", verb="GET"} | 1.248660241
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/authentication.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.099412211
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/authorization.k8s.io/v1?timeout=32s", verb="GET"} | 0.002372744
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/authorization.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.299374821
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/autoscaling/v1?timeout=32s", verb="GET"} | 1.19871405
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/autoscaling/v2beta1?timeout=32s", verb="GET"} | 1.448753136
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/autoscaling/v2beta2?timeout=32s", verb="GET"} | 1.148790586
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/batch/v1?timeout=32s", verb="GET"} | 0.699155697
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/batch/v1beta1?timeout=32s", verb="GET"} | 0.998960333
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/certificates.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002329583
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/config.istio.io/v1alpha2?timeout=32s", verb="GET"} | 0.19947485
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/coordination.k8s.io/v1?timeout=32s", verb="GET"} | 0.00255896
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/coordination.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.149677943
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/crd.k8s.amazonaws.com/v1alpha1?timeout=32s", verb="GET"} | 0.549296545
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/crd.projectcalico.org/v1?timeout=32s", verb="GET"} | 0.002270277
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/custom.metrics.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.012747798
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.349364577
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/dynatrace.com/v1alpha1?timeout=32s", verb="GET"} | 0.001302129
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/elbv2.k8s.aws/v1alpha1?timeout=32s", verb="GET"} | 0.049350027
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/elbv2.k8s.aws/v1beta1?timeout=32s", verb="GET"} | 0.002372812
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/events.k8s.io/v1beta1?timeout=32s", verb="GET"} | 1.348550366
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/extensions/v1beta1?timeout=32s", verb="GET"} | 0.002311416
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/hpa.orkaproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.002432777
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/iammanager.keikoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.002576066
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/install.istio.io/v1alpha1?timeout=32s", verb="GET"} | 0.002513564
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/instancemgr.keikoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.649275446
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/litmuschaos.io/v1alpha1?timeout=32s", verb="GET"} | 0.749110148
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/metrics.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.951508069
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/monitoring.coreos.com/v1?timeout=32s", verb="GET"} | 0.002462623
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/monitoring.coreos.com/v1alpha1?timeout=32s", verb="GET"} | 1.049249703
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/networking.istio.io/v1alpha3?timeout=32s", verb="GET"} | 1.498888937
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/networking.istio.io/v1beta1?timeout=32s", verb="GET"} | 0.002323638
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/networking.k8s.io/v1?timeout=32s", verb="GET"} | 0.002411563
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/networking.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.249401094
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/node.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.849161453
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/policy/v1beta1?timeout=32s", verb="GET"} | 0.899041388
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/rbac.authorization.k8s.io/v1?timeout=32s", verb="GET"} | 0.002423248
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/rbac.authorization.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002327157
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/rbac.istio.io/v1alpha1?timeout=32s", verb="GET"} | 0.002481619
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/scheduling.k8s.io/v1?timeout=32s", verb="GET"} | 0.00236329
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/scheduling.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002318702
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/security.istio.io/v1beta1?timeout=32s", verb="GET"} | 0.002456169
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/storage.k8s.io/v1?timeout=32s", verb="GET"} | 0.002295293
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/storage.k8s.io/v1beta1?timeout=32s", verb="GET"} | 1.3985861019999999
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/upgrademgr.keikoproj.io/v1alpha1/namespaces/%7Bnamespace%7D/rollingupgrades/%7Bname%7D/status", verb="PUT"} | 91.31436567799945
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/upgrademgr.keikoproj.io/v1alpha1/rollingupgrades?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D", verb="GET"} | 0.01303986
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/upgrademgr.keikoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.449370049
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/vpcresources.k8s.aws/v1beta1?timeout=32s", verb="GET"} | 0.799149227
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis?timeout=32s", verb="GET"}

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • rolling-upgrade-controller version
  • Kubernetes version :
$ kubectl version -o yaml

Other debugging information (if applicable):

  • RollingUpgrade status:
$ kubectl describe rollingupgrade <rollingupgrade-name>
  • controller logs:
$ kubectl logs <rolling-upgrade-controller pod>
@shaoxt
Copy link
Collaborator Author

shaoxt commented Aug 27, 2021

metrics_iks1.txt

Most of the metrics are from rest_client_request_latency_seconds

grep rest_client_request_latency_seconds metrics_iks1.txt |wc -l
769

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@shaoxt