Istio howto broken

[magware-dev]

I installed ambassador via Helm and followed the Istio HowTo.

Helm Chart version: ambassador-6.7.13
Isitio version: 1.10.3

I followed the instructions here: https://www.getambassador.io/docs/edge-stack/latest/howtos/istio/#integrating-productname-with-istio-15-and-above

The ambassador deployment YAML that includes the istio envoy proxy that is used on that page doesn't work with the deployment that is installed in the Helm Chart.

The first error prevents that the deployment is applied at all: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"service":"ambassador"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

The deployment from the Helm Chart has different selectors:

spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: ambassador
      app.kubernetes.io/name: ambassador

After changing the selectors, the YAML can be applied, and after a restart the ambassador pod boots and is healthy, but it doesn't work. I'm not sure what is going on, but I think the problem is in the istio-envoy container.

I managed to create a working version of the deployment by basically comparing the deployment from the Helm Chart and the one from the HowTo:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ambassador
  namespace: ambassador
  labels:
    app.kubernetes.io/instance: ambassador
    app.kubernetes.io/name: ambassador
    app.kubernetes.io/part-of: ambassador
    product: aes
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: ambassador
      app.kubernetes.io/name: ambassador
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8877"
        prometheus.io/scheme: "http"
        prometheus.io/path: "/metrics"
      labels:
        app.kubernetes.io/instance: ambassador
        app.kubernetes.io/name: ambassador
        product: aes
    spec:
      volumes:
        - name: istio-certs
          emptyDir:
            medium: Memory
        - name: istiod-ca-cert
          configMap:
            defaultMode: 420
            name: istio-ca-root-cert
        - emptyDir:
            medium: Memory
          name: istio-envoy
        - name: istio-token
          projected:
            defaultMode: 420
            sources:
              - serviceAccountToken:
                  audience: istio-ca
                  expirationSeconds: 43200
                  path: istio-token
        - downwardAPI:
            defaultMode: 420
            items:
              - fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.labels
                path: labels
          name: ambassador-pod-info
        - name: ambassador-edge-stack-secrets
          secret:
            secretName: ambassador-edge-stack
            defaultMode: 420
      containers:
      - name: ambassador
        image: docker.io/datawire/aes:1.13.10
        imagePullPolicy: IfNotPresent
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        env:
        - name: AMBASSADOR_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: REDIS_URL
          value: ambassador-redis:6379
        - name: HOST_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
        - name: AMBASSADOR_ISTIO_SECRET_DIR
          value: "/etc/istio-certs"
        - name: AMBASSADOR_ENVOY_BASE_ID
          value: "1"
        livenessProbe:
          httpGet:
            path: /ambassador/v0/check_alive
            port: admin
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 1
          periodSeconds: 3
          successThreshold: 1
          failureThreshold: 3
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 8443
          name: https
          protocol: TCP
        - containerPort: 8877
          name: admin
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /ambassador/v0/check_ready
            port: admin
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 1
          periodSeconds: 3
          successThreshold: 1
          failureThreshold: 3
        resources:
          limits:
            cpu: '1'
            memory: 600Mi
          requests:
            cpu: 200m
            memory: 300Mi
        securityContext:
          allowPrivilegeEscalation: false
        volumeMounts:
        - mountPath: /tmp/ambassador-pod-info
          name: ambassador-pod-info
        - mountPath: /.config/ambassador
          name: ambassador-edge-stack-secrets
          readOnly: true
        - mountPath: /etc/istio-certs/
          name: istio-certs
      - name: istio-proxy
        # Use the same version as your Istio installation
        image: istio/proxyv2:1.10.3
        args:
        - proxy
        - sidecar
        - --domain
        - $(POD_NAMESPACE).svc.cluster.local
        - --serviceCluster
        - istio-proxy-ambassador
        - --discoveryAddress
        - istio-pilot.istio-system.svc:15012
        - --connectTimeout
        - 10s
        - --statusPort
        - "15020"
        - --trust-domain=cluster.local
        - --controlPlaneBootstrap=false
        env:
        - name: OUTPUT_CERTS
          value: "/etc/istio-certs"
        - name: JWT_POLICY
          value: third-party-jwt
        - name: PILOT_CERT_PROVIDER
          value: istiod
        - name: CA_ADDR
          value: istiod.istio-system.svc:15012
        - name: ISTIO_META_MESH_ID
          value: cluster.local
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: INSTANCE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: SERVICE_ACCOUNT
          valueFrom:
            fieldRef:
              fieldPath: spec.serviceAccountName
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: ISTIO_META_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: ISTIO_META_CONFIG_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: ISTIO_META_CLUSTER_ID
          value: Kubernetes
        imagePullPolicy: IfNotPresent
        readinessProbe:
          failureThreshold: 30
          httpGet:
            path: /healthz/ready
            port: 15020
            scheme: HTTP
          initialDelaySeconds: 1
          periodSeconds: 2
          successThreshold: 1
          timeoutSeconds: 1
        volumeMounts:
        - mountPath: /var/run/secrets/istio
          name: istiod-ca-cert
        - mountPath: /etc/istio/proxy
          name: istio-envoy
        - mountPath: /etc/istio-certs/
          name: istio-certs
        - mountPath: /var/run/secrets/tokens
          name: istio-token
        securityContext:
          runAsUser: 0
      restartPolicy: Always
      securityContext:
        runAsUser: 8888
      serviceAccountName: ambassador
      terminationGracePeriodSeconds: 0
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

I changed lots of things and I'm not sure what change made it work, but my version is basically a copy of the one from the Helm Chart, with the exception that I added the prometheus.io annotations which allows the prometheus operator to configure prometheus without having to change the ConfigMap manually how it is described later in the HowTo.

Since I don't really know what I'm doing, I didn't create a PR here (also for some reason I cannot find the istio HowTo page in this repository). But maybe my deployment YAML helps someone to update that HowTo.

If not then this is just a "bug" report, that the Istio HowTo doesn't work.