panic: runtime error: invalid memory address or nil pointer dereference

[recollir]

We get the following error directly after startup:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1a496fc]

Kube-metrics-adapter is started with the following config:

    spec:
      serviceAccountName: custom-metrics-apiserver
      containers:
      - name: kube-metrics-adapter
        image: registry.opensource.zalan.do/teapot/kube-metrics-adapter:v0.1.5
        args:
        - --v=9
        - --prometheus-server=http://prometheus.kube-system.svc.cluster.local

We see the following in the logs:

❯ kubectl logs -f kube-metrics-adapter-76d4b79d96-79ddb
I1023 12:55:30.771682       1 serving.go:307] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I1023 12:55:30.772011       1 dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I1023 12:55:31.677636       1 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kube-metrics-adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJ ... Bzg4KqvToPew" 'https://100.64.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication'
I1023 12:55:31.684293       1 round_trippers.go:443] GET https://100.64.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 6 milliseconds
I1023 12:55:31.684309       1 round_trippers.go:449] Response Headers:
I1023 12:55:31.684312       1 round_trippers.go:452]     Cache-Control: no-cache, private
I1023 12:55:31.684315       1 round_trippers.go:452]     Content-Type: application/json
I1023 12:55:31.684318       1 round_trippers.go:452]     Content-Length: 2756
I1023 12:55:31.684321       1 round_trippers.go:452]     Date: Fri, 23 Oct 2020 12:55:31 GMT
I1023 12:55:31.684323       1 round_trippers.go:452]     Audit-Id: 409c03d0-667a-4a16-8fff-cb0c0b48e0ed
I1023 12:55:31.684361       1 request.go:1068] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"a1e1aced-f52d-42ca-94dc-fc3ca907dc81","resourceVersion":"38","creationTimestamp":"2020-03-19T13:56:44Z"},"data":{"client-ca-file":"-----BEGIN CERTIFICATE-----\nMIIC0zCC ... 6ZS8hQ==\n-----END CERTIFICATE-----\n","requestheader-allowed-names":"[\"aggregator\"]","requestheader-client-ca-file":"-----BEGIN CERTIFICATE-----\nMIIC7TC ... BufjA\n-----END CERTIFICATE-----\n","requestheader-extra-headers-prefix":"[\"X-Remote-Extra-\"]","requestheader-group-headers":"[\"X-Remote-Group\"]","requestheader-username-headers":"[\"X-Remote-User\"]"}}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1a496fc]

goroutine 1 [running]:
github.com/zalando-incubator/kube-metrics-adapter/pkg/server.AdapterServerOptions.RunCustomMetricsAdapterServer(0xc00085c360, 0x0, 0x0, 0x101, 0x7ffc3a70a1f5, 0x33, 0x0, 0x0, 0x0, 0x0, ...)
	/workspace/pkg/server/start.go:129 +0x11c
github.com/zalando-incubator/kube-metrics-adapter/pkg/server.NewCommandStartAdapterServer.func1(0xc00025a000, 0xc0004268c0, 0x0, 0x2, 0x0, 0x0)
	/workspace/pkg/server/start.go:69 +0x85
github.com/spf13/cobra.(*Command).execute(0xc00025a000, 0xc00018c010, 0x2, 0x2, 0xc00025a000, 0xc00018c010)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:838 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc00025a000, 0xc000184120, 0x0, 0x0)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:943 +0x336
github.com/spf13/cobra.(*Command).Execute(...)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:883
main.main()
	/workspace/main.go:40 +0xb8

Kubernetes Version: 1.18.9

[mikkeloscar]

Can you try the latest version? v0.1.9 (We didn't mark it as a release in github which is maybe a bit confusing)

[recollir]

That worked. Cool. Thank you. It starts up.

I do now see this though:

I1023 15:02:48.090581       1 authorization.go:73] Forbidden: "/apis/external.metrics.k8s.io/v1beta1", Reason: ""
I1023 15:02:48.090610       1 authorization.go:73] Forbidden: "/apis/custom.metrics.k8s.io/v1beta1", Reason: ""

Should I be worried? The applied RBAC for the metrics adapter is from https://github.com/zalando-incubator/kube-metrics-adapter/blob/master/docs/rbac.yaml

[mikkeloscar]

Is kube-metrics-adapter running in the kube-system namespace or somewhere else? If not in kube-system you need a different configuration as discussed in #142

[recollir]

kube-system

[recollir]

I redeployed the rbac (deleting it first) and the deployment again. Now the error msg disappeared. We are fine. Thanks again.