Routes
Sometimes, you might notice errors when you try to access your services over a route.
-
Try to send a request to your destination with verbose mode and look for error messages. For example, you might curl the path to your app on the ingress gateway.
curl -vik https://${INGRESS_GW_IP}:443/ratings/1 -H "Host: www.example.com"
Example output:
Message Description Steps to resolve Connection reset by peer
The destination sent a reset ( RST
) packet and dropped the connection. No TLS handshake was established. This issue often indicates a TLS issue.Skip to Step 5 to check the virtual gateway configuration for the tls
section. Make sure the secret exists with valid credentials.Connection refused
Your gateway might not be listening on the right port. Skip to Step 5 to verify that the gateway listens on the correct port for the correct host. successfully set certificate...SSL_ERROR_SYSCALL in connection
Your TLS secrets are set up, but the hostname might not match the hosts in your virtual gateway configuration. For example, the secret might be a wildcard *.example.com
, but the virtual gateway configuration specifies onlywww.example.com
.Make sure that the TLS secrets for the gateway are in the same namespace as the ingress gateway. Then, make sure that the hostname that the secret is for matches the hostnames in the virtual gateway configuration. -
Verify that your Gloo setup is working correctly.
- Check the management server.
- Check the agent for the clusters where your resources run.
- Make sure that your management server and agent run the same minor version of Gloo.
meshctl version --kubecontext ${MGMT_CONTEXT} meshctl version --kubecontext ${REMOTE_CONTEXT1} meshctl version --kubecontext ${REMOTE_CONTEXT2}
-
Verify that your Istio gateway pods are in a
Running
state.kubectl get pods -A --context ${REMOTE_CONTEXT1}
Example output:
NAMESPACE NAME READY STATUS RESTARTS AGE istio-system istio-eastwestgateway-6559bbc949-xrfz6 1/1 Running 0 4d14h istio-system istio-ingressgateway-64c544cfff-jgnkp 1/1 Running 0 4d16h
If not, describe the pods, get the logs, and look for error messages.
kubectl describe pod -n gloo-mesh-gateways --context ${REMOTE_CONTEXT1} -l istio=ingressgateway
kubectl logs --context ${REMOTE_CONTEXT1} -n gloo-mesh-gateways $(kubectl get pods -l "app=istio-ingressgateway" -n gloo-mesh-gateways -o jsonpath='{.items[].metadata.name}') > logs-ingressgateway.txt
For example, a
filter_chain_not_found
message indicates that the request does not have a matching SNI in the gateway. Make sure that the TLS secrets for the gateway are in the same namespace as the ingress gateway. Then, make sure that the hostname that the secret is for matches the hostnames in the virtual gateway configuration. -
If you are sending a request directly to the gateway and not to a host, check the service details for the IP address that the gateway is exposed on. Verify that you use this IP address in your request to the route.
kubectl get svc -n istio-system --context ${REMOTE_CONTEXT1}
In the following example, the external IP addresses are as follows:
istio-eastwestgateway
gateway for traffic between cluster or in a service mesh:35.xxx.xx.x1
istio-ingressgateway
gateway for traffic from outside the cluster if you use Gloo Gateway with Gloo Mesh Enterprise:35.xxx.x.xx9
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE istio-eastwestgateway LoadBalancer 10.xxx.x.xx8 35.xxx.xx.x1 15021:30974/TCP,15443:30522/TCP 5d18h istio-ingressgateway LoadBalancer 10.xxx.xx.xx3 35.xxx.x.xx9 15021:31225/TCP,80:30381/TCP,443:31078/TCP,15443:30649/TCP 5d18h
-
If you are sending the request to a host, check your Gloo virtual gateway resource. Look for the information in the following table.
kubectl get virtualgateways -A -o yaml --context ${REMOTE_CONTEXT1}
Example output:
Review the following table to understand this configuration.apiVersion: networking.gloo.solo.io/v2 kind: VirtualGateway metadata: annotations: cluster.solo.io/cluster: "" name: istio-ingressgateway namespace: bookinfo spec: listeners: - allowedRouteTables: - host: www.example.com http: {} port: number: 443 tls: mode: SIMPLE secretName: example-secret workloads: - selector: labels: istio: ingressgateway
Setting Description namespace
Make sure that the gateway's namespace is in the same workspace as the route's app. Or, the gateway workspace must import the route's app. allowedRouteTables
Make sure that the host that you call is allowed. port
Check that the port matches the port that you call. For example, if you sent a request along an https://
URL, but the port is80
(for HTTP), the request fails.tls
Check the TLS details. If the gateway listens on port 443
for HTTPS traffic, this section is required.workloads
Make sure that the virtual gateway selects the gateway workload that you checked earlier. -
If you are sending the request to a host, check your Gloo route table resource. Look for the information in the following table.
kubectl get routetables -A -o yaml --context ${REMOTE_CONTEXT1}
Example output:
Review the following table to understand this configuration.apiVersion: networking.gloo.solo.io/v2 kind: RouteTable metadata: annotations: cluster.solo.io/cluster: "" name: www-example-com namespace: bookinfo spec: defaultDestination: port: number: 9080 ref: name: ratings namespace: bookinfo hosts: - www.example.com http: - forwardTo: {} labels: "no": auth matchers: - headers: - name: noauth value: "true" name: ratings-ingress-no-auth - forwardTo: {} labels: route: ratings name: ratings-ingress virtualGateways: - name: istio-ingressgateway
Setting Description namespace
Make sure that the route table's namespace is in the same workspace as the route's app. Or, the route table's workspace must import the route's app. defaultDestination
Make sure that the details for your app are correct, such as the port and reference. host
Verify that the host you call is included in the route table. forwardTo
Check the forward to details. For example, the route table might be set up to forward requests with certain headers or labels to a different service. virtualGateways
Make sure that the route table selects the virtual gateway that you checked earlier. -
Check for other Gloo policies that might be applied to the destination or route.
kubectl get ratelimitserverconfigs,RatelimitConfigs,ratelimitserversettings,ratelimitclientconfigs,ratelimitpolicies,wasmdeploymentpolicies,externalendpoints,externalservices,accesslogpolicies,failoverpolicies,faultinjectionpolicies,outlierdetectionpolicies,jwtpolicies,wafpolicies,retrytimeoutpolicies,accesspolicies,corspolicies,csrfpolicies,extauthpolicies,mirrorpolicies,transformationpolicies,authorizationpolicies,extauthserver,headermanipulationpolicies -A --context ${REMOTE_CONTEXT1}
-
Check that the underlying Istio resources are working properly. For example, your gateway might be missing a secret.
istioctl analyze --context ${REMOTE_CONTEXT1} -n istio-system