Routes

Sometimes, you might notice errors when you try to access your services over a route.

1. Try to send a request to your destination with verbose mode and look for error messages. For example, you might curl the path to your app on the ingress gateway.

     curl -vik --resolve www.example.com:80:${INGRESS_GW_IP} http://www.example.com:80/ratings/1
     

     curl -vik --resolve www.example.com:443:${INGRESS_GW_IP} https://www.example.com:443/ratings/1
     

   Example output:

   Message	Description	Steps to resolve
   Connection reset by peer	The destination sent a reset (RST) packet and dropped the connection. No TLS handshake was established. This issue often indicates a TLS issue.	Skip to Step 5 to check the virtual gateway configuration for the tls section. Make sure the secret exists with valid credentials.
   Connection refused	Your gateway might not be listening on the right port.	Skip to Step 5 to verify that the gateway listens on the correct port for the correct host.
   successfully set certificate...SSL_ERROR_SYSCALL in connection	Your TLS secrets are set up, but the hostname might not match the hosts in your virtual gateway configuration. For example, the secret might be a wildcard *.example.com, but the virtual gateway configuration specifies only www.example.com.	Make sure that the TLS secrets for the gateway are in the same namespace as the ingress gateway. Then, make sure that the hostname that the secret is for matches the hostnames in the virtual gateway configuration.
   403 with message forbidden: User \"system:anonymous\" cannot get path	The request might be resolving to the IP address of a cluster that is not configured for this domain.	Check that your DNS provider resolves the domain to the IP address of the ingress gateway in the correct cluster.
   HTTP 403, HTTPS 404 errors	The request is received, but you get an unexpected 403 forbidden on HTTP routes and 404 not found on HTTPS. You might have a policy configuration issue.	Check the ingress gateway logs to confirm that the request is received. Next, check the host values in the route tables and selected virtual gateway to confirm that the host matches the host that the request calls. Common errors include a mismatch such as example.com vs. www.example.com in the configuration files or in the request path. Then, check the policies that are applied to your route, such as described in a later command.

2. Verify that your Gloo setup is working correctly.

   1. Check the management server.
   2. Check the agent for the clusters where your resources run.
   3. Make sure that your management server and agent run the same minor version of Gloo.

        meshctl version --kubecontext ${MGMT_CONTEXT}
      meshctl version --kubecontext ${REMOTE_CONTEXT1}
      meshctl version --kubecontext ${REMOTE_CONTEXT2}
        

3. Verify that your Istio gateway pods are in a Running state. Note that the gateways may be deployed in a different namespace.

     kubectl get pods -A -n gloo-mesh-gateways --context ${REMOTE_CONTEXT1}
     

   Example output:

     NAME                                     READY   STATUS    RESTARTS   AGE
   istio-eastwestgateway-6559bbc949-xrfz6   1/1     Running   0          4d14h
   istio-ingressgateway-64c544cfff-jgnkp    1/1     Running   0          4d16h
     

   If not, describe the pods, get the logs, and look for error messages.

     kubectl describe pod -n gloo-mesh-gateways --context ${REMOTE_CONTEXT1} -l istio=ingressgateway
     

     kubectl logs --context ${REMOTE_CONTEXT1} -n gloo-mesh-gateways $(kubectl get pods -l "app=istio-ingressgateway" -n gloo-mesh-gateways -o jsonpath='{.items[].metadata.name}') > logs-ingressgateway.txt
     

   For example, a filter_chain_not_found message indicates that the request does not have a matching SNI in the gateway. Make sure that the TLS secrets for the gateway are in the same namespace as the ingress gateway. Then, make sure that the hostname that the secret is for matches the hostnames in the virtual gateway configuration.

4. If you are sending a request directly to the gateway and not to a host, check the service details for the IP address that the gateway is exposed on. Verify that you use this IP address in your request to the route.

     kubectl get svc -n istio-system --context ${REMOTE_CONTEXT1}
     

   In the following example, the external IP addresses are as follows:

   • istio-eastwestgateway gateway for traffic between cluster or in a service mesh: 35.xxx.xx.x1
   • istio-ingressgateway gateway for traffic from outside the cluster if you use Gloo Mesh Gateway with Gloo Mesh Enterprise: 35.xxx.x.xx9

     NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                                                      AGE
   istio-eastwestgateway   LoadBalancer   10.xxx.x.xx8    35.xxx.xx.x1   15021:30974/TCP,15443:30522/TCP                              5d18h
   istio-ingressgateway    LoadBalancer   10.xxx.xx.xx3   35.xxx.x.xx9   15021:31225/TCP,80:30381/TCP,443:31078/TCP,15443:30649/TCP   5d18h
     

5. If you are sending the request to a host, check your Gloo virtual gateway resource. Look for the information in the following table.

     kubectl get virtualgateways -A -o yaml --context ${REMOTE_CONTEXT1}
     

   Example output:

     apiVersion: networking.gloo.solo.io/v2
   kind: VirtualGateway
   metadata:
     annotations:
       cluster.solo.io/cluster: ""
     name: istio-ingressgateway
     namespace: bookinfo
   spec:
     listeners:
     - allowedRouteTables:
       - host: www.example.com
       http: {}
       port:
         number: 443
       tls:
         mode: SIMPLE
         secretName: example-secret
     workloads:
     - selector:
         labels:
           istio: ingressgateway
     

   Review the following table to understand this configuration.

   Setting	Description
   namespace	Make sure that the gateway’s namespace is in the same workspace as the route’s app. Or, the gateway workspace must import the route’s app.
   allowedRouteTables	Make sure that the host that you call is allowed. Common errors include a mismatch such as example.com vs. www.example.com in the configuration files or in the request path.
   port	Check that the port matches the port that you call. For example, if you sent a request along an https:// URL, but the port is 80 (for HTTP), the request fails.
   tls	Check the TLS details. If the gateway listens on port 443 for HTTPS traffic, this section is required.
   workloads	Make sure that the virtual gateway selects the gateway workload that you checked earlier.

6. If you are sending the request to a host, check your Gloo route table resource. Look for the information in the following table.

     kubectl get routetables -A -o yaml --context ${REMOTE_CONTEXT1}
     

   Example output:

     apiVersion: networking.gloo.solo.io/v2
   kind: RouteTable
   metadata:
     annotations:
       cluster.solo.io/cluster: ""
     name: www-example-com
     namespace: bookinfo
   spec:
     defaultDestination:
       port:
         number: 9080
       ref:
         name: ratings
         namespace: bookinfo
     hosts:
     - www.example.com
     http:
     - forwardTo: {}
       labels:
         "no": auth
       matchers:
       - headers:
         - name: noauth
           value: "true"
       name: ratings-ingress-no-auth
     - forwardTo: {}
       labels:
         route: ratings
       name: ratings-ingress
     virtualGateways:
     - name: istio-ingressgateway
     

   Review the following table to understand this configuration.

   Setting	Description
   namespace	Make sure that the route table’s namespace is in the same workspace as the route’s app. Or, the route table’s workspace must import the route’s app.
   defaultDestination	Make sure that the details for your app are correct, such as the port and reference.
   host	Verify that the host you call is included in the route table. Common errors include a mismatch such as example.com vs. www.example.com in the configuration files or in the request path.
   forwardTo	Check the forward to details. For example, the route table might be set up to forward requests with certain headers or labels to a different service.
   virtualGateways	Make sure that the route table selects the virtual gateway that you checked earlier.

7. Check for other Gloo policies that might be applied to the destination or route.

     kubectl get ratelimitserverconfigs,RatelimitConfigs,ratelimitserversettings,ratelimitclientconfigs,ratelimitpolicies,wasmdeploymentpolicies,externalendpoints,externalservices,accesslogpolicies,failoverpolicies,faultinjectionpolicies,outlierdetectionpolicies,jwtpolicies,wafpolicies,retrytimeoutpolicies,accesspolicies,corspolicies,csrfpolicies,extauthpolicies,mirrorpolicies,transformationpolicies,authorizationpolicies,extauthserver,headermanipulationpolicies -A --context ${REMOTE_CONTEXT1}
     

   Example common policy configuration issues:

   • An external auth policy uses booleanExpr to control the order in which auth configurations such as JWT or basic auth are evaluated. However, the policy does not include the names of those auth configurations.
   • An external auth policy uses an OPA server to evaluate requests. However, the OPA server does not have any OPA config. After creating a config map with the OPA config, you must restart the external auth service and OPA server for the config to take effect.

8. Check that the underlying Istio resources are working properly. For example, your gateway might be missing a secret.

     istioctl analyze --context ${REMOTE_CONTEXT1} -n istio-system