ELB health checks in AWS fail
Debug why the ELB health checks in AWS are failing for the ingress gateway service.
What's happening?
You successfully installed Gloo Gateway in your cluster that is hosted on Amazon Web Services (AWS). However, the health check for the Elastic Load Balancer (ELB) that is automatically created for you to back the Kubernetes service that exposes the Istio ingress gateway, fails and reports an unhealthy state.
Why it's happening?
When you install Gloo Gateway, an Istio ingress gateway is automatically deployed for you and exposed with a Kubernetes service of type load balancer. In clusters that are hosted on AWS, an ELB is automatically created to back the ingress gateway service. Gloo Gateway configures the ingress gateway to listen on HTTPS port 15443. However, when the ELB is created, the first port that is defined in the Kubernetes service manifest is used to perform the ELB health check. This port might be different from the port that Gloo Gateway configures.
For example, the following Kubernetes service manifest defines multiple ports that the ingress gateway can listens on. Because port 80 is the first port that is defined in this list, the ELB health check is configured for port 80 instead of port 15443.
...
spec:
clusterIP: 10.100.108.166
externalTrafficPolicy: Cluster
ports:
- name: http2
nodePort: 31143
port: 80
protocol: TCP
targetPort: 8080
- name: https
nodePort: 30131
port: 443
protocol: TCP
targetPort: 8443
- name: tls
nodePort: 32287
port: 15443
protocol: TCP
targetPort: 15443
selector:
app: istio-ingressgateway
istio: ingressgateway
How to fix it?
For your ELB health check to pass, you need to configure the load balancer to run the health check on port 15443.
-
Edit the
istio-ingressgateway
load balancer service in your cluster.kubectl edit svc/istio-ingressgateway -n istio-system
-
Move the
tls
port to the top of the list of ports, such as in this example YAML file.... spec: clusterIP: 10.100.108.166 externalTrafficPolicy: Cluster ports: - name: tls nodePort: 32287 port: 15443 protocol: TCP targetPort: 15443 - name: http2 nodePort: 31143 port: 80 protocol: TCP targetPort: 8080 - name: https nodePort: 30131 port: 443 protocol: TCP targetPort: 8443 selector: app: istio-ingressgateway istio: ingressgateway
-
Save your changes.
-
Wait a few minutes and then verify that the ELB health checks pass successfully.