Debug your setup
Use built-in tools to troubleshoot issues in your Gloo Gateway setup.
Gloo Gateway is based on Envoy proxy. If you experience issues in your environment, such as policies that are not applied or traffic that is not routed correctly, in a lot of cases, these errors can be observed at the proxy. In this guide you learn how to use the Gloo Gateway and Envoy debugging tools to troubleshoot misconfigurations on the gateway.
Before you begin
If you have not done yet, install the glooctl
CLI. The glooctl
CLI is a convenient tool that helps you gather important information about your gateway proxy. To install the glooctl
, you run the following command:
curl -sL https://run.solo.io/gloo/install | sh
export PATH=$HOME/.gloo/bin:$PATH
Make sure to use the version of glooctl
that matches your installed version, such as .
Debug your Gloo Gateway setup
Check the Gloo Gateway installation. You can do that by using the
glooctl check
command that quickly checks the health of Gloo Gateway deployments, pods, and custom resources, and verifies Gloo resource configuration. Any issues that are found are reported back in the CLI output.glooctl check
Example output for a misconfigured VirtualHostOption:
Found rejected VirtualHostOption by 'gloo-system': gloo-system jwt (Reason: 2 errors occurred: * invalid virtual host [http~bookinfo_example] while processing plugin enterprise_warning: Could not load configuration for the following Enterprise features: [jwt]
Get the details for the failed resource. For example to get the details of a VirtualHostOption, you can use the following command.
kubectl get virtualhostoption <name> -n <namespace>
If the resources seem to be ok, you can check the configuration that is applied on your gateway. Configuration might be missing on the gateway or might be applied to the wrong route. For example, if you apply multiple RouteOption resources to the same route by using the
targetRefs
section, only the oldest RouteOption is applied. The newer RouteOption configuration is ignored and not applied to the gateway.glooctl get proxy -o yaml
If the proxy resource seems to be ok, you can access the debugging interface of your gateway proxy on your localhost.
kubectl port-forward deploy/gloo-proxy-http -n gloo-system 19000 &
Common endpoints that can help troubleshoot your setup further, include:
Endpoint Description config_dump Get the configuration that is available in the Envoy proxy. Any Gloo Gateway resources that you create are translated in to Envoy configuration. Depending on whether or not you enabled resource validation, you might have applied invalid configuration that is rejected Envoy. You can also use glooctl proxy dump
to get the Envoy proxy configuration.listeners See the listeners that are configured on your gateway. logging Review the log level that is set for each component. stats/prometheus View metrics that Envoy emitted and sent to the built-in Prometheus instance. Check the proxy configuration that is served by the Gloo Gateway xDS server. When you create Gloo Gateway resources, these resources are translated into Envoy configuration and sent to the xDS server. If Gloo Gateway resources are configured correctly, the configuration must be included in the proxy configuration that is served by the xDS server.
glooctl proxy served-config --name gloo-proxy-http
Review the components of your Gloo Gateway setup. The number of components varies depending on whether you installed the Open source or Enterprise Edition of Gloo Gateway.
kubectl get pods -n gloo-system
Example output for Open source:
NAME READY STATUS RESTARTS AGE discovery-857796b8fb-gcphh 1/1 Running 0 15h gateway-proxy-8689c55fb8-7swfq 1/1 Running 0 15h gloo-66fb8974c9-8sgll 1/1 Running 0 15h
Example output for Enterprise Edition:
NAME READY STATUS RESTARTS AGE discovery-68dbd794-ssx7b 1/1 Running 0 107m extauth-67557744dd-5wc2p 1/1 Running 0 107m gateway-proxy-79c9f44b5d-cprg7 1/1 Running 0 107m gloo-74bb8b9df7-72t8m 1/1 Running 0 107m gloo-fed-857964dd9f-gq8np 1/1 Running 0 107m gloo-fed-console-6f99dddccd-ls64k 3/3 Running 0 107m glooe-grafana-865bb9cd45-cdshq 1/1 Running 0 107m glooe-prometheus-kube-state-metrics-v2-55ffc89cbb-kr8jx 1/1 Running 0 107m glooe-prometheus-server-7d5b85764c-2nl2w 2/2 Running 0 107m observability-5f8ffc8bdc-zggxb 1/1 Running 0 107m rate-limit-6d66688567-5tcx8 1/1 Running 3 107m redis-57fd559c5c-hcd6n 1/1 Running 0 107m
Review the logs for each component. Each component logs the sync loops that it runs, such as syncing with various environment signals like the Kubernetes API. You can fetch the latest logs for all the components with the following command.
glooctl debug logs # save the logs to a file glooctl debug logs -f gloo.log # only print errors glooctl debug logs --errors-only
You can use the
kubectl logs
command to view logs for individual components.kubectl logs -f -n gloo-system -l gloo=gloo
To follow the logs of other Gloo Gateway components, simply change the value of the
gloo
label as shown in the table below.Component Command Discovery kubectl logs -f -n gloo-system -l gloo=discovery
External Auth (Enterprise) kubectl logs -f -n gloo-system -l gloo=extauth
Gloo Control Plane kubectl logs -f -n gloo-system -l gloo=gloo
Observability (Enterprise) kubectl logs -f -n gloo-system -l gloo=observability
Rate Limiting (Enterprise) kubectl logs -f -n gloo-system -l gloo=rate-limit
If you still cannot troubleshoot the issue, capture the logs and the state of Gloo Gateway in a file.
glooctl debug logs -f gloo-logs.log glooctl debug yaml -f gloo-yamls.yaml
Choose between the following options to get help from Solo engineers.
- For Open source users: Join the Solo Slack to chat with a Solo engineer directly.
- For Enterprise users: Join the Solo Slack or open a support ticket.