ELB health checks in AWS fail

Debug why the ELB health checks in AWS are failing for the ingress gateway service.

What's happening?

You successfully installed Gloo Gateway in your cluster that is hosted on Amazon Web Services (AWS). However, the health check for the Elastic Load Balancer (ELB) that is automatically created for you to back the Kubernetes service that exposes the Istio ingress gateway, fails and reports an unhealthy state.

Why it's happening?

When you install Gloo Gateway, an Istio ingress gateway is automatically deployed for you and exposed with a Kubernetes service of type load balancer. In clusters that are hosted on AWS, an ELB is automatically created to back the ingress gateway service. Gloo Gateway configures the ingress gateway to listen on HTTPS port 15443. However, when the ELB is created, the first port that is defined in the Kubernetes service manifest is used to perform the ELB health check. This port might be different from the port that Gloo Gateway configures.

For example, the following Kubernetes service manifest defines multiple ports that the ingress gateway can listens on. Because port 80 is the first port that is defined in this list, the ELB health check is configured for port 80 instead of port 15443.

...
spec:
  clusterIP: 10.100.108.166
  externalTrafficPolicy: Cluster
  ports:
  - name: http2
    nodePort: 31143
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 30131
    port: 443
    protocol: TCP
    targetPort: 8443
  - name: tls
    nodePort: 32287
    port: 15443
    protocol: TCP
    targetPort: 15443
  selector:
    app: istio-ingressgateway
    istio: ingressgateway

How to fix it?

For your ELB health check to pass, you need to configure the load balancer to run the health check on port 15443.

  1. Edit the istio-ingressgateway load balancer service in your cluster.

    kubectl edit svc/istio-ingressgateway -n istio-system
    
  2. Move the tls port to the top of the list of ports, such as in this example YAML file.

    ...
    spec:
      clusterIP: 10.100.108.166
      externalTrafficPolicy: Cluster
      ports:
      - name: tls
        nodePort: 32287
        port: 15443
        protocol: TCP
        targetPort: 15443
      - name: http2
        nodePort: 31143
        port: 80
        protocol: TCP
        targetPort: 8080
      - name: https
        nodePort: 30131
        port: 443
        protocol: TCP
        targetPort: 8443
      selector:
        app: istio-ingressgateway
        istio: ingressgateway
    
  3. Save your changes.

  4. Wait a few minutes and then verify that the ELB health checks pass successfully.