Use built-in tools to troubleshoot issues in your Gloo Gateway setup.

Gloo Gateway is based on Envoy proxy. If you experience issues in your environment, such as policies that are not applied or traffic that is not routed correctly, in a lot of cases, these errors can be observed at the proxy. In this guide you learn how to use the Gloo Gateway and Envoy debugging tools to troubleshoot misconfigurations on the gateway.

Debug the control plane

  1. Enable port-forwarding on the control plane.

      kubectl port-forward deploy/gloo-gateway -n gloo-system 9095

  2. In your browser, open the admin server debugging interface: http://localhost:9095/.

    Figure: Admin server debugging interface.
    Figure: Admin server debugging interface.

  3. Select one of the endpoints to continue debugging. Review the following table to understand this configuration.

    EndpointDescription
    /debug/pprofView the pprof profile of the control plane. A profile shows you the stack traces of the call sequences, such as Go routines, that led to particular events, such as memory allocation. The endpoint includes descriptions of each available profile.
    /loggingReview the current logging levels of each component in the control plane. You can also interactively set the log level by component, such as to enable DEBUG logs.
    /snapshots/krtView the current krt snapshot, or the point-in-time view of the transformed Kubernetes resources and their sync status that the control plane processed. These resources are then used to generate gateway configuration that is sent to the gateway proxies for routing decisions.
    /snapshots/xdsView the current xDS snapshot, or the Envoy-specific configuration (such as Listeners, Routes, Backends, and Workloads) that is being sent to and applied by Envoy gateway proxies. These snapshots show the final translated configuration that Envoy gateway proxies use for routing decisions.

Debug your gateway setup

  1. Make sure that the Gloo Gateway control plane and gateway proxies are running. For any pod that is not running, describe the pod for more details.

      kubectl get pods -n gloo-system

  2. Check the HTTPRoutes for the status of the route and any attached policies.

      kubectl get httproutes -A
  
      kubectl get httproute <name> -n <namespace> -o yaml

  3. Access the debugging interface of your gateway proxy on your localhost. Configuration might be missing on the gateway or might be applied to the wrong route. For example, if you apply multiple policies to the same route by using the targetRefs section, only the oldest policy is applied. The newer policy configuration might be ignored and not applied to the gateway.

      kubectl port-forward deploy/http -n gloo-system 19000 &
    Figure: Debugging interface of the gateway proxy.
    Figure: Debugging interface of the gateway proxy.

    Common endpoints that can help troubleshoot your setup further, include:

    EndpointDescription
    config_dumpGet the configuration that is available in the Envoy proxy. Any kgateway resources that you create are translated in to Envoy configuration. Depending on whether or not you enabled resource validation, you might have applied invalid configuration that is rejected in Envoy. You can also use glooctl proxy dump to get the Envoy proxy configuration.
    listenersSee the listeners that are configured on your gateway.
    loggingReview the log level that is set for each component.
    stats/prometheusView metrics that Envoy emitted and sent to the built-in Prometheus instance.
      kubectl port-forward deploy/agentgateway -n gloo-system 15000 &

    Open your browser to the following endpoints.

    EndpointDescription
    http://localhost:15000/config_dumpGet the configuration that is available in the agentgateway proxy. Any custom resources that you create are translated in to agentgateway configuration. Depending on whether or not you enabled resource validation, you might have applied invalid configuration that is rejected in agentgateway.
    http://localhost:15000/uiA read-only user interface to review the agentgateway resources in your environment, such as listeners, routes, backends, and policies.
    Figure: Read-only agentgateway UI.
    Figure: Read-only agentgateway UI.

  4. Review the logs for each component. Each component logs the sync loops that it runs, such as syncing with various environment signals like the Kubernetes API. You can fetch the latest logs for all the components with the following command.

      # Gloo Gateway control plane
kubectl logs -n gloo-system deployment/gloo-gateway

# Replace $GATEWAY_NAME with the name of your gateway.
export GATEWAY_NAME=http
kubectl logs -n gloo-system deployment/$GATEWAY_NAME

Set gateway proxy debug logging

You can set the log level for the Envoy proxy to get more detailed logs. Envoy log level options include trace, debug, info, warn, error, critical, and off. The default log level is info. For more information, see Debugging Envoy.

  1. Create a GatewayParameters resource to add any custom settings to the gateway. For other settings, see the GatewayParameters API docs or check out the Gateway customization guides.

      kubectl apply -f- <<EOF
apiVersion: gateway.kgateway.dev/v1alpha1
kind: GatewayParameters
metadata:
  name: debug-gateway
  namespace: gloo-system
spec:
  kube: 
    envoyContainer:
      bootstrap:
        logLevel: debug
EOF

  2. Create a Gateway resource that references your custom GatewayParameters.

      kubectl apply -f- <<EOF
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: debug-gateway
  namespace: gloo-system
spec:
  gatewayClassName: gloo-gateway-v2
  infrastructure:
    parametersRef:
      name: debug-gateway
      group: gateway.kgateway.dev
      kind: GatewayParameters       
  listeners:
  - protocol: HTTP
    port: 80
    name: http
    allowedRoutes:
      namespaces:
        from: All
EOF

  3. Verify that a pod is created for your gateway proxy and that it has the pod settings that you defined in the GatewayParameters resource.

      kubectl get pods -l app.kubernetes.io/name=debug-gateway -n gloo-system -o yaml

  4. Create an HTTPRoute that routes traffic to your app through the debug gateway. The following example assumes that you set up the sample httpbin app.

      kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
  namespace: httpbin
spec:
  parentRefs:
    - name: debug-gateway
      namespace: gloo-system
  hostnames:
    - "debug.com"
  rules:
    - backendRefs:
        - name: httpbin
          port: 8000
EOF

  5. Get the address of the debug gateway proxy.

      export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-system debug-gateway -o=jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS
  
      kubectl port-forward deployment/debug-gateway -n gloo-system 8080:8080

  6. Send traffic through the debug gateway proxy.

      curl -i http://$INGRESS_GW_ADDRESS:8080/headers -H "host: debug.com:8080"
  
      curl -i localhost:8080/headers -H "host: debug.com"

  7. Review the debug logs for the gateway proxy.

      kubectl logs -n gloo-system -l app.kubernetes.io/name=debug-gateway

    Example output:

      [2025-07-08 18:59:13.234][34][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:254] [Tags: "ConnectionId":"2"] destroying stream: 0 active remaining, readyForStream false, currentUnusedCapacity 1
[2025-07-08 18:59:14.240][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:774] [Tags: "ConnectionId":"1"] remote close
[2025-07-08 18:59:14.241][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:314] [Tags: "ConnectionId":"1"] closing socket: 0
[2025-07-08 18:59:14.243][34][debug][conn_handler] [external/envoy/source/common/listener_manager/active_stream_listener_base.cc:136] [Tags: "ConnectionId":"1"] adding to cleanup list
[2025-07-08 18:59:14.244][1][debug][main] [external/envoy/source/server/server.cc:245] flushing stats
[2025-07-08 18:59:18.232][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:774] [Tags: "ConnectionId":"2"] remote close
[2025-07-08 18:59:18.233][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:314] [Tags: "ConnectionId":"2"] closing socket: 0
[2025-07-08 18:59:18.233][34][debug][client] [external/envoy/source/common/http/codec_client.cc:107] [Tags: "ConnectionId":"2"] disconnect. resetting 0 pending requests
[2025-07-08 18:59:18.234][34][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:532] [Tags: "ConnectionId":"2"] client disconnected, failure reason: 
[2025-07-08 18:59:18.235][34][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:500] invoking 1 idle callback(s) - is_draining_for_deletion_=false

TrafficPolicy not applied

As part of debugging, you might have noticed that your HTTPRoute or Gateway had an attached TrafficPolicy. The TrafficPolicy’s status might say Accepted and seem normal. However, when you checked the gateway configuration, the policy is not applied to the selected routes. Review the following common reasons for missing policies.

  1. Verify that the TrafficPolicy is attached correctly. For example, you might use label selectors that do not match any HTTPRoute or Gateway. For more information, see Policy attachment.

  2. Confirm that you do not have multiple, conflicting policies. In general, the oldest policy is enforced. For more information, see Policy priority and merging rules.

  3. Determine if you need a Kubernetes ReferenceGrant. For example, the TrafficPolicy might rely on a GatewayExtension to enable a feature such as external auth. However, the GatewayExtension might be in a different namespace than the backing external auth service.

    Example ReferenceGrant for external auth GatewayExtension:

    • The GrantExtension for external auth, HTTPRoute, and backing Service are in the app namespace, such as httpbin.
    • The external auth service is in the gloo-system namespace.
      apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: reference-grant
  namespace: gloo-system
spec:
  from:
    - group: gateway.kgateway.dev
      kind: GatewayExtension
      namespace: httpbin
  to:
    - group: ""
      kind: Service