Service templates do not seem to work

[mariusstaicu]

On operator version 2.0.2 with emqx 5.0.12, it seems like the listenerServiceTemplate does not do anything.

  listenersServiceTemplate:
    metadata:
      name: emqx-listeners
    spec:
      ports:
        - name: mqtt
          port: 1883
          protocol: TCP
          targetPort: 1883
        - name: mqtts
          port: 8883
          protocol: TCP
          targetPort: 8883

Also, cannot change the name of the default listener via bootstrap config.
Also I don't think there's a way of changing the emqx-dashboard service config, becauase adding a dashboardServiceTemplate on port 18083 shows port already in use.

The EMQX "emqx" is invalid: spec.dashboardServiceTemplate.spec.ports[1]: Duplicate value: map[string]interface {}{"port":18083, "protocol":"TCP"}

[Gala-R]

On operator version 2.0.2 with emqx 5.0.12, it seems like the listenerServiceTemplate does not do anything.

  listenersServiceTemplate:
    metadata:
      name: emqx-listeners
    spec:
      ports:
        - name: mqtt
          port: 1883
          protocol: TCP
          targetPort: 1883
        - name: mqtts
          port: 8883
          protocol: TCP
          targetPort: 8883

Also, cannot change the name of the default listener via bootstrap config. Also I don't think there's a way of changing the emqx-dashboard service config, becauase adding a dashboardServiceTemplate on port 18083 shows port already in use.

The EMQX "emqx" is invalid: spec.dashboardServiceTemplate.spec.ports[1]: Duplicate value: map[string]interface {}{"port":18083, "protocol":"TCP"}

Can you show your CR， I want to know your configuration. @mariusstaicu

[mariusstaicu]

apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
  name: emqx
spec:
  image: emqx/emqx:5.0.12
  bootstrapConfig: |
    authentication = [
      {
        auto_reconnect = true
        mechanism = "password_based"
        backend = "postgresql"
        database = "creds-service"
        username = "creds-service"
        password_hash_algorithm {name = "plain", salt_position = "disable"}
        pool_size = 1
        query = "SELECT password, is_superuser FROM device_credentials where username = ${username} LIMIT 1"
        server = "127.0.0.1:5432"
        ssl {enable = false, verify = "verify_peer"}
      }
    ]
    authorization {
      cache {
        enable = true
        max_size = 32
        ttl = "5m"
      }
      deny_action = ignore
      no_match = "deny"
      sources = [
        {
          enable = true
          type = "postgresql"
          server = "127.0.0.1:5432"
          database = "creds-service"
          username = "creds-service"
          auto_reconnect = true
          pool_size = 1
          query = "SELECT case when access = 1 then 'subscribe' when access = 2 then 'publish' when access = 3 then 'all' end as action, case when allow = 1 then 'allow' else 'deny' end as permission, topic FROM device_acl where username = ${username}"
          ssl {
            enable = false
          }
        }
      ]
    }
    listeners {
      tcp {
        default {
          bind = "0.0.0.0:1883"
          max_connections = 1024000
        }
      }
      ssl {
        mqtts {
          bind = "0.0.0.0:8883"
    
          max_connections = 102400
    
          ssl_options {
            keyfile = "/etc/certs/tls.key"
            certfile = "/etc/certs/tls.crt"
            versions = ["tlsv1.3", "tlsv1.2", "tlsv1.1", "tlsv1"]
          }
        }
      }
    }
  coreTemplate:
    metadata:
      name: emqx-core
      labels:
        apps.emqx.io/instance: emqx
        apps.emqx.io/db-role: core
    spec:
      initContainers:
        - name: sysctl-buddy
          image: busybox:1.31
          securityContext:
            privileged: true
          command: [ "/bin/sh" ]
          args:
            - -c
            - sysctl -w fs.file-max=2097152 fs.nr_open=2097152 net.core.somaxconn=32768 net.ipv4.tcp_max_syn_backlog=16384 net.netfilter.nf_conntrack_max=1000000 net.netfilter.nf_conntrack_tcp_timeout_time_wait=30 net.ipv4.tcp_max_tw_buckets=1048576 net.ipv4.tcp_fin_timeout=15
          resources:
            requests:
              cpu: 10m
              memory: 10Mi
            limits:
              cpu: 10m
              memory: 10Mi
      extraContainers:
        - name: cloudsql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.30.1
          command: [ "/cloud_sql_proxy",
                     "-instances=compelling-cat-212513:europe-west2:integration=tcp:5432",
                     "-credential_file=/secrets/cloudsql/cloud-sql-service-account.json" ]
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          volumeMounts:
            - name: cloud-sql-service-account
              mountPath: /secrets/cloudsql
              readOnly: true
      replicas: 2
      env:
        - name: EMQX_AUTHENTICATION__1__PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
        - name: EMQX_AUTHORIZATION__SOURCES__1__PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
      volumeClaimTemplates:
        resources:
          requests:
            storage: 1Gi
        accessModes:
          - ReadWriteOnce
      livenessProbe:
        httpGet:
          path: /status
          port: 18083
        initialDelaySeconds: 60
        periodSeconds: 30
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /status
          port: 18083
        initialDelaySeconds: 10
        periodSeconds: 5
        failureThreshold: 12
      resources:
        requests:
          cpu: 200m
          memory: 600Mi
        limits:
          cpu: 600m
          memory: 1Gi
      podSecurityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        fsGroupChangePolicy: Always
      containerSecurityContext:
        runAsUser: 1000
        runAsGroup: 1000
      extraVolumeMounts:
        - name: emqx-certs
          mountPath: /etc/certs
          readOnly: true
      extraVolumes:
        - name: emqx-certs
          secret:
            secretName: emqx-certificate
        - name: cloud-sql-service-account
          secret:
            secretName: cloud-sql-service-account
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: apps.emqx.io/instance
                      operator: In
                      values:
                        - emqx
                topologyKey: "kubernetes.io/hostname"
              weight: 1
  replicantTemplate:
    spec:
      replicas: 0
  listenersServiceTemplate:
    metadata:
      name: emqx-listeners
    spec:
      ports:
      - name: mqtt
        port: 1883
        protocol: TCP
        targetPort: 1883
      - name: mqtts
        port: 8883
        protocol: TCP
        targetPort: 8883
  dashboardServiceTemplate:
    metadata:
      name: emqx-dashboard
    spec:
      ports:
        - name: dashboard
          port: 18083
          protocol: TCP
          targetPort: 18083

[Gala-R]

the default list

Thank you for your reply. I probably understand your question. I will verify it as soon as possible and give you an answer.

apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
  name: emqx
spec:
  image: emqx/emqx:5.0.12
  bootstrapConfig: |
    authentication = [
      {
        auto_reconnect = true
        mechanism = "password_based"
        backend = "postgresql"
        database = "creds-service"
        username = "creds-service"
        password_hash_algorithm {name = "plain", salt_position = "disable"}
        pool_size = 1
        query = "SELECT password, is_superuser FROM device_credentials where username = ${username} LIMIT 1"
        server = "127.0.0.1:5432"
        ssl {enable = false, verify = "verify_peer"}
      }
    ]
    authorization {
      cache {
        enable = true
        max_size = 32
        ttl = "5m"
      }
      deny_action = ignore
      no_match = "deny"
      sources = [
        {
          enable = true
          type = "postgresql"
          server = "127.0.0.1:5432"
          database = "creds-service"
          username = "creds-service"
          auto_reconnect = true
          pool_size = 1
          query = "SELECT case when access = 1 then 'subscribe' when access = 2 then 'publish' when access = 3 then 'all' end as action, case when allow = 1 then 'allow' else 'deny' end as permission, topic FROM device_acl where username = ${username}"
          ssl {
            enable = false
          }
        }
      ]
    }
    listeners {
      tcp {
        default {
          bind = "0.0.0.0:1883"
          max_connections = 1024000
        }
      }
      ssl {
        mqtts {
          bind = "0.0.0.0:8883"
    
          max_connections = 102400
    
          ssl_options {
            keyfile = "/etc/certs/tls.key"
            certfile = "/etc/certs/tls.crt"
            versions = ["tlsv1.3", "tlsv1.2", "tlsv1.1", "tlsv1"]
          }
        }
      }
    }
  coreTemplate:
    metadata:
      name: emqx-core
      labels:
        apps.emqx.io/instance: emqx
        apps.emqx.io/db-role: core
    spec:
      initContainers:
        - name: sysctl-buddy
          image: busybox:1.31
          securityContext:
            privileged: true
          command: [ "/bin/sh" ]
          args:
            - -c
            - sysctl -w fs.file-max=2097152 fs.nr_open=2097152 net.core.somaxconn=32768 net.ipv4.tcp_max_syn_backlog=16384 net.netfilter.nf_conntrack_max=1000000 net.netfilter.nf_conntrack_tcp_timeout_time_wait=30 net.ipv4.tcp_max_tw_buckets=1048576 net.ipv4.tcp_fin_timeout=15
          resources:
            requests:
              cpu: 10m
              memory: 10Mi
            limits:
              cpu: 10m
              memory: 10Mi
      extraContainers:
        - name: cloudsql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.30.1
          command: [ "/cloud_sql_proxy",
                     "-instances=compelling-cat-212513:europe-west2:integration=tcp:5432",
                     "-credential_file=/secrets/cloudsql/cloud-sql-service-account.json" ]
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          volumeMounts:
            - name: cloud-sql-service-account
              mountPath: /secrets/cloudsql
              readOnly: true
      replicas: 2
      env:
        - name: EMQX_AUTHENTICATION__1__PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
        - name: EMQX_AUTHORIZATION__SOURCES__1__PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
      volumeClaimTemplates:
        resources:
          requests:
            storage: 1Gi
        accessModes:
          - ReadWriteOnce
      livenessProbe:
        httpGet:
          path: /status
          port: 18083
        initialDelaySeconds: 60
        periodSeconds: 30
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /status
          port: 18083
        initialDelaySeconds: 10
        periodSeconds: 5
        failureThreshold: 12
      resources:
        requests:
          cpu: 200m
          memory: 600Mi
        limits:
          cpu: 600m
          memory: 1Gi
      podSecurityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        fsGroupChangePolicy: Always
      containerSecurityContext:
        runAsUser: 1000
        runAsGroup: 1000
      extraVolumeMounts:
        - name: emqx-certs
          mountPath: /etc/certs
          readOnly: true
      extraVolumes:
        - name: emqx-certs
          secret:
            secretName: emqx-certificate
        - name: cloud-sql-service-account
          secret:
            secretName: cloud-sql-service-account
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: apps.emqx.io/instance
                      operator: In
                      values:
                        - emqx
                topologyKey: "kubernetes.io/hostname"
              weight: 1
  replicantTemplate:
    spec:
      replicas: 0
  listenersServiceTemplate:
    metadata:
      name: emqx-listeners
    spec:
      ports:
      - name: mqtt
        port: 1883
        protocol: TCP
        targetPort: 1883
      - name: mqtts
        port: 8883
        protocol: TCP
        targetPort: 8883
  dashboardServiceTemplate:
    metadata:
      name: emqx-dashboard
    spec:
      ports:
        - name: dashboard
          port: 18083
          protocol: TCP
          targetPort: 18083

Thank you for your reply. I probably understand your question. I will verify it as soon as possible and give you an answer.

[Gala-R]

On operator version 2.0.2 with emqx 5.0.12, it seems like the listenerServiceTemplate does not do anything.

  listenersServiceTemplate:
    metadata:
      name: emqx-listeners
    spec:
      ports:
        - name: mqtt
          port: 1883
          protocol: TCP
          targetPort: 1883
        - name: mqtts
          port: 8883
          protocol: TCP
          targetPort: 8883

Also, cannot change the name of the default listener via bootstrap config. Also I don't think there's a way of changing the emqx-dashboard service config, becauase adding a dashboardServiceTemplate on port 18083 shows port already in use.

The EMQX "emqx" is invalid: spec.dashboardServiceTemplate.spec.ports[1]: Duplicate value: map[string]interface {}{"port":18083, "protocol":"TCP"}

1. When you configure listeners in bootstrapConfig, you don’t need to configure it repeatedly in the listener service. The Operator will get the listener enabled by EMQX and write it into the listener service.
2. The Operator will configure a default Dashboard listener, the name is: dashboard-listeners-http-bind, and the port is: 18083, because you have configured the dashboard service, and the Port is also 18083, which is the same as the default value configured by the Operator, so you appear mentioned that error.

[mariusstaicu]

1. Ok, but listeners via bootstrap config is one thing and services exposing the listeners to the outside is other thing. I thought listenerServiceTemplate is used to customize the way the k8s service looks and bootstrap config refers to the internal listener configuration of emqx. I don't think they should be linked together.
   Also, regarding your answer, if I only configure listeners via bootstrapConfig, there will be no kubernetes svc created for 1883 and 8883 ports. I have to create the svc manually. Currently I am creating a service resource manually, because the operator does not create one, no matter if I add listenerServiceTemplate or not to the EMQX resource.

2. Same with the dashboardServiceTemplate. I thought it is used to customize the svc resource.
   If I want to make the operator give a different service name to the emqx-dashboard service is there a way to do it from the EMQX resource?

[Gala-R]

1. Ok, but listeners via bootstrap config is one thing and services exposing the listeners to the outside is other thing. I thought listenerServiceTemplate is used to customize the way the k8s service looks and bootstrap config refers to the internal listener configuration of emqx. I don't think they should be linked together.
   Also, regarding your answer, if I only configure listeners via bootstrapConfig, there will be no kubernetes svc created for 1883 and 8883 ports. I have to create the svc manually. Currently I am creating a service resource manually, because the operator does not create one, no matter if I add listenerServiceTemplate or not to the EMQX resource.
2. Same with the dashboardServiceTemplate. I thought it is used to customize the svc resource.
   If I want to make the operator give a different service name to the emqx-dashboard service is there a way to do it from the EMQX resource?

When you configure listeners in bootstrapConfig, Operator will automatically discover these enabled listeners and write them into the service。As for the dashboard listener, the Operator will be enabled by default, and will also be written into the service. If you define a listener in bootstrapConfig and configure a listener in the service, when the names of the two are different, the Operator will fail to create the service.