Gloo Gateway is based on Envoy proxy. If you experience issues in your environment, such as policies that are not applied or traffic that is not routed correctly, in a lot of cases, these errors can be observed at the proxy. In this guide you learn how to use the Gloo Gateway and Envoy debugging tools to troubleshoot misconfigurations on the gateway.

Before you begin

If you have not done yet, install the glooctl CLI. The glooctl CLI is a convenient tool that helps you gather important information about your gateway proxy. To install the glooctl, you run the following command:

  curl -sL https://run.solo.io/gloo/install | sh
export PATH=$HOME/.gloo/bin:$PATH

Debug your Gloo Gateway setup

  1. Check the Gloo Gateway installation. You can do that by using the glooctl check command that quickly checks the health of Gloo Gateway deployments, pods, and custom resources, and verifies Gloo resource configuration. Any issues that are found are reported back in the CLI output.

      glooctl check

    Example output for a misconfigured VirtualHostOption:

      Found rejected VirtualHostOption by 'gloo-system': gloo-system jwt (Reason: 2 errors occurred:
 * invalid virtual host [http~bookinfo_example] while processing plugin enterprise_warning: Could not load configuration for the following Enterprise features: [jwt]

  2. Get the details for the failed resource. For example to get the details of a VirtualHostOption, you can use the following command.

      kubectl get virtualhostoption <name> -n <namespace>

  3. If the resources seem to be ok, you can check the configuration that is applied on your gateway. Configuration might be missing on the gateway or might be applied to the wrong route. For example, if you apply multiple RouteOption resources to the same route by using the targetRefs section, only the oldest RouteOption is applied. The newer RouteOption configuration is ignored and not applied to the gateway.

      glooctl get proxy -o yaml

  4. If the proxy resource seems to be ok, you can access the debugging interface of your gateway proxy on your localhost.

      kubectl port-forward deploy/gloo-proxy-http -n gloo-system 19000 &
    Figure: Debugging interface of the gateway proxy.

    Common endpoints that can help troubleshoot your setup further, include:

    EndpointDescription
    config_dumpGet the configuration that is available in the Envoy proxy. Any Gloo Gateway resources that you create are translated in to Envoy configuration. Depending on whether or not you enabled resource validation, you might have applied invalid configuration that is rejected Envoy. You can also use glooctl proxy dump to get the Envoy proxy configuration.
    listenersSee the listeners that are configured on your gateway.
    loggingReview the log level that is set for each component.
    stats/prometheusView metrics that Envoy emitted and sent to the built-in Prometheus instance.

  5. Check the proxy configuration that is served by the Gloo Gateway xDS server. When you create Gloo Gateway resources, these resources are translated into Envoy configuration and sent to the xDS server. If Gloo Gateway resources are configured correctly, the configuration must be included in the proxy configuration that is served by the xDS server.

      glooctl proxy served-config --name gloo-proxy-http

  6. Review the components of your Gloo Gateway setup. The number of components varies depending on whether you installed the Open source or Enterprise Edition of Gloo Gateway.

      kubectl get pods -n gloo-system

    Example output for Open source:

      NAME                             READY   STATUS    RESTARTS   AGE
discovery-857796b8fb-gcphh       1/1     Running   0          15h
gateway-proxy-8689c55fb8-7swfq   1/1     Running   0          15h
gloo-66fb8974c9-8sgll            1/1     Running   0          15h

    Example output for Enterprise Edition:

      NAME                                                      READY   STATUS    RESTARTS   AGE
discovery-68dbd794-ssx7b                                  1/1     Running   0          107m
extauth-67557744dd-5wc2p                                  1/1     Running   0          107m
gateway-proxy-79c9f44b5d-cprg7                            1/1     Running   0          107m
gloo-74bb8b9df7-72t8m                                     1/1     Running   0          107m
gloo-fed-857964dd9f-gq8np                                 1/1     Running   0          107m
gloo-fed-console-6f99dddccd-ls64k                         3/3     Running   0          107m
glooe-grafana-865bb9cd45-cdshq                            1/1     Running   0          107m
glooe-prometheus-kube-state-metrics-v2-55ffc89cbb-kr8jx   1/1     Running   0          107m
glooe-prometheus-server-7d5b85764c-2nl2w                  2/2     Running   0          107m
observability-5f8ffc8bdc-zggxb                            1/1     Running   0          107m
rate-limit-6d66688567-5tcx8                               1/1     Running   3          107m
redis-57fd559c5c-hcd6n                                    1/1     Running   0          107m

  7. Review the logs for each component. Each component logs the sync loops that it runs, such as syncing with various environment signals like the Kubernetes API. You can fetch the latest logs for all the components with the following command.

      glooctl debug logs
# save the logs to a file
glooctl debug logs -f gloo.log
# only print errors
glooctl debug logs --errors-only

    You can use the kubectl logs command to view logs for individual components.

      kubectl logs -f -n gloo-system -l gloo=gloo

    To follow the logs of other Gloo Gateway components, simply change the value of the gloo label as shown in the table below.

    ComponentCommand
    Discoverykubectl logs -f -n gloo-system -l gloo=discovery
    External Auth (Enterprise)kubectl logs -f -n gloo-system -l gloo=extauth
    Gloo Control Planekubectl logs -f -n gloo-system -l gloo=gloo
    Observability (Enterprise)kubectl logs -f -n gloo-system -l gloo=observability
    Rate Limiting (Enterprise)kubectl logs -f -n gloo-system -l gloo=rate-limit

  8. If you still cannot troubleshoot the issue, capture the logs and the state of Gloo Gateway in a file.

      glooctl debug logs -f gloo-logs.log
glooctl debug yaml -f gloo-yamls.yaml

  9. Choose between the following options to get help from Solo engineers.