Otel-operator does not create clusterrole/ and clusterrole binding for otel-collector #1679

krimeshshah · 2023-04-26T17:48:48Z

Hi Team,

After deploying otel-operator , when i deployed otel-collector as descibed on this page https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-operator, i noticed that serviceaccount otel-collector was created but corresponding clusterrole and clusterrolebinding is not created by operator. Shouldnt it be taken care by operator.? Since it is not able to create clusterrole and clusterrolebinding we face issue fetching metrics from pods across different namespace

#######################################################################

E0415 19:49:43.196149 1 reflector.go:140] k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
W0415 19:50:30.471358 1 reflector.go:424] k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
###########################################################################

Regards,
Krimesh

TylerHelmuth · 2023-04-26T17:53:03Z

Original issue: open-telemetry/opentelemetry-helm-charts#762

krimeshshah · 2023-04-27T11:37:02Z

@TylerHelmuth @Allex1 Any work around i can use. Because using otel-col as a separate chart will not solve my purpose of scraping service monitor metrics. It has to be deployed with otel operator. Also as pe this issue open-telemetry/opentelemetry-helm-charts#69 - one work around mentioned is creating CR inside template directory by @VineethReddy02 . Any idea how do i do it?

TylerHelmuth · 2023-04-27T14:35:27Z

At the moment there is no community helm chart for managing the custom resources themselves, you have to maintain them yourself. I believe most users do that by managing the yaml file itself since it is only 1 file.

As for the clusterrole/clusterrolbindings, how these are handled will be dependent on how your cluster is managed. Your cluster managers could create and maintain the resources for you and the collector managed by the operator could use them. I still like the idea of the operator supporting the creation of those resources when it creates the collector.

krimeshshah · 2023-04-27T15:45:46Z

@TylerHelmuth These are the steps that i did to have the otel-cllector managed by operator

Installed otel-operator with helm chart that also installed opentelemetrycollectors.opentelemetry.io crd.
Then as per the instruction on operator page -https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-operator . i created otel-collector
otel-collector was deployed successfully but getting below error while accessing pods at clusterscop to get the metrics

########################################################################
ng:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
W0427 15:00:13.754890 1 reflector.go:424] k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
E0427 15:00:13.754925 1 reflector.go:140] k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
W0427 15:00:50.934078 1 reflector.go:424] k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
E0427 15:00:50.934105 1 reflector.go:140] k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
W0427 15:01:31.769279 1 reflector.go:424] k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
E0427 15:01:31.769323 1 reflector.go:140] k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
W0427 15:02:03.341587 1 reflector.go:424] k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
E0427 15:02:03.341616 1 reflector.go:140] k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
W0427 15:02:39.106772 1 reflector.go:424] k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
E0427 15:02:39.106800 1 reflector.go:140] k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
W0427 15:03:29.794105 1 reflector.go:424] k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:logging:otelcol-collector" cannot list resource "pods" in API group "" at the cluster scope
E0427 15:0
##############################################################################

To fix this issue i created cluster role and clusterrole biding for the otel-collector service account.

krimeshshah · 2023-04-27T15:46:32Z

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: otel-collector
labels:
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
rules:

apiGroups: [""] # "" indicates the core API group
resources: ["pod"]
verbs: ["get", "watch", "list"]

#################################################
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: otelcol
subjects:

kind: ServiceAccount
name: otelcol-collector # name of your service account
namespace: logging #v this is the namespace your service account is in
roleRef: # referring to your ClusterRole
kind: ClusterRole
name: otel-collector
apiGroup: rbac.authorization.k8s.io
#####################################################

krimeshshah · 2023-04-27T15:47:39Z

Shouldnt this fix the issue? or is this the right way to add or apply changes to crd managed by operator? Considering creation of otel-collector, clusterrole and clusterrolebinding is the change on top of crd

TylerHelmuth · 2023-04-27T18:09:20Z

That does fix the issue, and is an acceptable solution. We discussed this issue in the Operator SIG meeting today and agreed that it is not the responsibility of the opentelemetry-operator chart since the chart does not manage an OpentelemetryCollector custom resource nor is it the responsibility of the operator (like I thought it would be).

Long term we think this chart would be a chart that is responsible for creating a clusterrole/clusterrolebinding for the collector created by the OpentelemetryCollector custom resource. This would follow the pattern used by other operator/helm chart combinations like kube-prometheus-stack.

paologallinaharbur · 2023-09-05T17:17:12Z

Hello, @TylerHelmuth I'd like just to confirm that the expected behaviour is that:

The helm chart of the otel collector creates Roles and bindings if needed by any of the presets. Es presets.kubeletMetrics.enabled=true adds the following template:

rules:
  - apiGroups: [""]
    resources: ["nodes/stats"]
    verbs: ["get", "watch", "list"]

On the other hand, the chart of the operator or the operator itself is not and it is not expected in the future to manage roles or bindings. Therefore, if any permission is needed by the service account, the user needs to add them "manually"

If this is confirmed I'd state it clearly in the readme.. I was a bit confused to realize the chart is "smarter" (covering more use-cases) than the operator.

If so I image that we can close this as not-planned/will-not-do

garett-eiq · 2023-09-11T16:04:26Z

I encountered this today and it's kind of confusing behavior. Am I correct in thinking the recommended approach is that in addition to instantiating an OpenTelemetryCollector custom resource via the OTEL operator, I also need to create/specify a service account for the collector I'm creating, along with the ClusterRole and ClusterRoleBinding?

If so:

i don't see any details about that anywhere in the README. can that be placed there so it's not a gotcha?
it sounds like this is considered out of scope for the operator, but I don't really understand why. the expected behavior (for me, and i assume many others) is that creating the OpenTelemetryCollector resource will create everything needed for it to run. it's very surprising that necessary permissions aren't created for that, or at least have some configurable options for how you'd like to set it up. this kind of seems like a blind spot in terms of usability and adoption. or maybe i'm missing something?

pavolloffay · 2023-11-27T13:34:28Z

done in #2327

thefirstofthe300 · 2024-01-10T20:31:12Z

@pavolloffay Can we reopen this issue? The PR you linked to fixed an orthogonal issue to the one described by this issue. The PR adds a missing permission to the OTEL operator's controller.

This issue is about having the operator controller manage RBAC for the deployed OTEL collector deployment/daemonset and the service account generated for that specific deployment/daemonset.

jaronoff97 · 2024-01-12T19:26:07Z

@thefirstofthe300 is the resolved by #2396 ?

thefirstofthe300 · 2024-01-13T06:13:07Z

I believe this issue does indeed get fixed by that PR. Nice to see just how fast that wait was. 😆

heruscode · 2024-04-19T03:28:06Z

Hi, does the above #2396 fix also affect receivers? Because I am trying to use the prometheus receiver and I still get some errors in the logs of the collector pod(created by OpenTelemetryCollector operator resource)

OpenTelemetryCollector config:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'emissary-ingress'
          scrape_interval: 30s
          kubernetes_sd_configs:
            - role: endpoints
          relabel_configs:
            - source_labels: [__meta_kubernetes_endpoints_label_service]
              action: keep
              regex: ambassador-admin
exporters:
  prometheus:
    endpoint: 0.0.0.0:9464
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [prometheus]

logs:

W0419 03:05:00.650568       1 reflector.go:539] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:emissary:emissary-ingress-collector" cannot list resource "endpoints" in API group "" at the cluster scope
E0419 03:05:00.650604       1 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:emissary:emissary-ingress-collector" cannot list resource "endpoints" in API group "" at the cluster scope
W0419 03:05:21.574835       1 reflector.go:539] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:emissary:emissary-ingress-collector" cannot list resource "services" in API group "" at the cluster scope
E0419 03:05:21.574869       1 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:emissary:emissary-ingress-collector" cannot list resource "services" in API group "" at the cluster scope

jaronoff97 · 2024-04-24T17:08:21Z

@iblancasa will this be fixed by #2787 ?

iblancasa · 2024-04-25T16:05:49Z

@iblancasa will this be fixed by #2787 ?

It is not related, I think. This seems to be a different problem.

jaronoff97 added enhancement New feature or request good first issue Good for newcomers question Further information is requested area:controller labels Apr 27, 2023

husnialhamdani mentioned this issue Nov 8, 2023

add missing pod in the rbac #2327

Merged

pavolloffay closed this as completed Nov 27, 2023

pavolloffay reopened this Jan 11, 2024

jaronoff97 closed this as completed Jan 13, 2024

iblancasa mentioned this issue Aug 27, 2024

Restrict Cluster Role access authorizations #3156

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Otel-operator does not create clusterrole/ and clusterrole binding for otel-collector #1679

Otel-operator does not create clusterrole/ and clusterrole binding for otel-collector #1679

krimeshshah commented Apr 26, 2023

TylerHelmuth commented Apr 26, 2023

krimeshshah commented Apr 27, 2023

TylerHelmuth commented Apr 27, 2023

krimeshshah commented Apr 27, 2023

krimeshshah commented Apr 27, 2023 •

edited

Loading

krimeshshah commented Apr 27, 2023

TylerHelmuth commented Apr 27, 2023

paologallinaharbur commented Sep 5, 2023 •

edited

Loading

garett-eiq commented Sep 11, 2023

pavolloffay commented Nov 27, 2023

thefirstofthe300 commented Jan 10, 2024

jaronoff97 commented Jan 12, 2024

thefirstofthe300 commented Jan 13, 2024

heruscode commented Apr 19, 2024

jaronoff97 commented Apr 24, 2024

iblancasa commented Apr 25, 2024

Otel-operator does not create clusterrole/ and clusterrole binding for otel-collector #1679

Otel-operator does not create clusterrole/ and clusterrole binding for otel-collector #1679

Comments

krimeshshah commented Apr 26, 2023

TylerHelmuth commented Apr 26, 2023

krimeshshah commented Apr 27, 2023

TylerHelmuth commented Apr 27, 2023

krimeshshah commented Apr 27, 2023

krimeshshah commented Apr 27, 2023 • edited Loading

krimeshshah commented Apr 27, 2023

TylerHelmuth commented Apr 27, 2023

paologallinaharbur commented Sep 5, 2023 • edited Loading

garett-eiq commented Sep 11, 2023

pavolloffay commented Nov 27, 2023

thefirstofthe300 commented Jan 10, 2024

jaronoff97 commented Jan 12, 2024

thefirstofthe300 commented Jan 13, 2024

heruscode commented Apr 19, 2024

jaronoff97 commented Apr 24, 2024

iblancasa commented Apr 25, 2024

krimeshshah commented Apr 27, 2023 •

edited

Loading

paologallinaharbur commented Sep 5, 2023 •

edited

Loading