Vault Service Registration Failing in Consul

[nikashnarula]

What did you do?
Received alerts for "Vault Service Registration Failed in Consul". Want to understand if this could be an issue with prometheus-consul-exporter as overall vault/consul health looks healthy.

What did you expect to see?
Vault Service to show a healthy state and show value of 1 in health metrics.

What did you see instead? Under which circumstances?

Environment
Production

• System information:

  Linux 5.4.0-1103-aws x86_64

• consul_exporter version:

  0.9.0

• Consul version:

  Consul v1.15.3
  Revision 7ce982ce
  Build Date 2023-06-01T20:40:32Z
  Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

• Prometheus version:

    /prometheus # prometheus --version
    prometheus, version 2.43.1 (branch: HEAD, revision: e278195e3983c966c2a0f42211f62fa8f40c5561)
      build user:       root@fdbae5f7538f
      build date:       20230504-20:56:42
      go version:       go1.19.9
      platform:         linux/amd64
      tags:             netgo,builtinassets
  

• Prometheus configuration file:

insert configuration here (if relevant to the issue)

• Logs:

`kubectl logs pod/prometheus-consul-exporter-857f4f4df8-8dgf2 -n cloudops -f
ts=2023-09-07T12:36:12.457Z caller=consul_exporter.go:79 level=info msg="Starting consul_exporter" version="(version=0.9.0, branch=HEAD, revision=3dc6d7b4de8f35235472f8faafe820e1c2dbc42f)"
ts=2023-09-07T12:36:12.457Z caller=consul_exporter.go:80 level=info build_context="(go=go1.19.3, user=root@6ca0da38f88e, date=20221129-13:48:45)"
ts=2023-09-07T12:36:12.458Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9107
ts=2023-09-07T12:36:12.458Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9107
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/cloudops-msk-zookeeper-humio?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/snapscheduler-metrics-cloudops?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/cam-nb-rest-csp?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/nb-rest-issues?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/azure-wi-webhook-webhook-service-azure-workload-identity-system?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/storage-fleet-resource-notify-fleet?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/pub-keystore-tunnel?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/grpc-apphost-appdm?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/cloudops-kube-eagle-cloudops?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/storage-fleet-resource-sync-util-fleet?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/kube-state-metrics-cloudops?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.554Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/istiod-istio-system?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.756Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/sc-ops-prometheus-elasticsearch-exporter-cdsops-nomesh?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/ui-sfm?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/consul-consul-ui-cloudops?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/cache-portal-mgr?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/storagecentral-humio-headless-cloudops?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/grpc-auth-tokens?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/es-connect-audit?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/k8s-inventory-grpc-csp?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/prometheus-adapter-cdsops-nomesh?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/storage-fleet-capabilities-fleet?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/reports-nb-rest-zerto?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/hci-hca-grpc-api-hci-manager?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/int-file-gql-data-graph-file-services?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/elasticsearch-master-headless-istio-system?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/nb-rest-virt-appdm?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/jaeger-query-istio-system?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/cadence-frontend-headless-workflow?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/pub-rest-backup-recovery-reports-appdm?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.756Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/fleet-gql-data-graph-fleet?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.757Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/cloudops-msk-humio?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.846Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/grpc-opemgr-appdm?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:11.853Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/inventory-grpc-csp?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:12.046Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/grpc-groups?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:12.047Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/ccs-portal-mgr?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:12.146Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/virtualization-manager-grpc-zerto?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:12.149Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/nb-rest-nimble-virt-manager?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:12.447Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/pub-rest-dashboard-appdm?stale=\": net/http: TLS handshake timeout"
ts=2023-10-05T10:04:12.447Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/shield-worker-intellistack?stale=\": net/http: TLS handshake timeout"
ts=2024-01-22T02:36:12.444Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-42ef2d8cb38409e5-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.444Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/default-ingress-nginx-controller-service-cloudops?stale=\": EOF"
ts=2024-01-22T02:36:12.445Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-69f1af8d004169ef-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.445Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-4cbbfb8c7549ece2-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.445Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/consul-consul-ui-cloudops?stale=\": EOF"
ts=2024-01-22T02:36:12.445Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-aa46d18c659fc986-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.445Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-2fe95e8cc36510d0-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.445Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-3ce7938d19de7afd-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.445Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-0db4cd8c6afd33a2-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.446Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/grpc-storeonce-manager?stale=\": EOF"
ts=2024-01-22T02:36:12.544Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/cam-nb-rest-csp?stale=\": EOF"
ts=2024-01-22T02:36:12.544Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/prometheus-k8s-pods-cloudops?stale=\": EOF"
ts=2024-01-22T02:36:12.545Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/grpc-dualauth?stale=\": EOF"
ts=2024-01-22T02:36:12.545Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-3fc1d68c7d0383ac-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.545Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/hci-setup-nb-rest-hci-manager?stale=\": EOF"
ts=2024-01-22T02:36:12.548Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-8bd4348d2205ef17-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.548Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-46049e8cbf88464b-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.548Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-26c90c8cadb8d6f7-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.644Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-92a54c8c9730dcdb-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.746Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-a1ba5c8d0ac01a6f-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.644Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-c40e7b8c690ecdbc-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:12.945Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-3e2ff38c8c78850b-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:13.445Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-7ef5e58c8a633669-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:13.744Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-69b6788c7ccc9758-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:13.845Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/dataprotection-nb-rest-cspdm?stale=\": EOF"
ts=2024-01-22T02:36:13.847Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/priv-object-tunnel?stale=\": EOF"
ts=2024-01-22T02:36:13.847Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/grpc-virt-appdm?stale=\": EOF"
ts=2024-01-22T02:36:13.847Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/grpc-rip-portal-mgr?stale=\": EOF"
ts=2024-01-22T02:36:13.848Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/virtualization-manager-nb-rest-zerto?stale=\": EOF"
ts=2024-01-22T02:36:13.851Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-5600ee8cd4c65c36-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:13.851Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-634c398d1ecdf081-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:13.852Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-c51adc8cb6851691-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:13.852Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-886c728cf6cadc27-driver-svc-intellistack?stale=\": EOF"
ts=2024-01-22T02:36:13.853Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/int-fleet-nb-rest-fleet?stale=\": EOF"
ts=2024-01-22T02:36:13.853Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/druid-historical-cs-druid?stale=\": EOF"
ts=2024-01-22T02:36:13.945Z caller=consul_exporter.go:406 level=error msg="Failed to query service health" err="Get \"https://consul-consul-server:8501/v1/health/service/spark-49255c8d10d574b2-driver-svc-intellistack?stale=\": EOF"`