Skip to content
home
2.0.x
Operations
You are viewing the documentation for the 2.0.x version of Gloo Gateway. To get the latest enterprise features of the Kubernetes Gateway API on top of the kgateway project, see Solo Enterprise for kgateway. To use the older Gloo Gateway 1.x that is based on the Gloo upstream project, click here.

Debug

Page as Markdown
Debug your gateway setup.
Use built-in tools to troubleshoot issues in your Gloo Gateway setup.
Gloo Gateway consists of the control plane and an Envoy-based kgateway data plane. If you experience issues in your environment, such as policies that are not applied or traffic that is not routed correctly, in a lot of cases, these errors can be observed at the proxy.
Debug the control plane

  1. Enable port-forwarding on the control plane.
    kubectl port-forward deploy/gloo-gateway -n gloo-system 9095
  2. In your browser, open the admin server debugging interface: http://localhost:9095/.
    Figure: Admin server debugging interface.
    Figure: Admin server debugging interface.
  3. Select one of the endpoints to continue debugging. Review the following table to understand this configuration.
    EndpointDescription
    /debug/pprofView the pprof profile of the control plane. A profile shows you the stack traces of the call sequences, such as Go routines, that led to particular events, such as memory allocation. The endpoint includes descriptions of each available profile.
    /loggingReview the current logging levels of each component in the control plane. You can also interactively set the log level by component, such as to enable DEBUG logs.
    /snapshots/krtView the current krt snapshot, or the point-in-time view of the transformed Kubernetes resources and their sync status that the control plane processed. These resources are then used to generate gateway configuration that is sent to the gateway proxies for routing decisions.
    /snapshots/xdsView the current xDS snapshot, or the Envoy-specific configuration (such as Listeners, Routes, Backends, and Workloads) that is being sent to and applied by Envoy gateway proxies. These snapshots show the final translated configuration that Envoy gateway proxies use for routing decisions. For snapshots to be present, you must have a Gateway resource that creates a connected, Envoy-based kgateway proxy.
Debug your gateway setup

  1. Make sure that the Gloo Gateway control plane and gateway proxies are running. For any pod that is not running, describe the pod for more details.
    kubectl get pods -n gloo-system
  2. Check the HTTPRoutes for the status of the route and any attached policies.
    kubectl get httproutes -A
    kubectl get httproute <name> -n <namespace> -o yaml
  3. Access the debugging interface of your gateway proxy on your localhost. Configuration might be missing on the gateway or might be applied to the wrong route. For example, if you apply multiple policies to the same route by using the targetRefs section, only the oldest policy is applied. The newer policy configuration might be ignored and not applied to the gateway.
  4. Review the logs for each component. Each component logs the sync loops that it runs, such as syncing with various environment signals like the Kubernetes API.
Set gateway proxy debug logging

You can set the log level for the Envoy proxy to get more detailed logs. Envoy log level options include trace, debug, info, warn, error, critical, and off. The default log level is info. For more information, see Debugging Envoy.
  1. Create a GlooGatewayParameters resource to add any custom settings to the gateway. For other settings, see the GlooGatewayParameters API docs or check out the Gateway customization guides.
    kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1alpha1
kind: GlooGatewayParameters
metadata:
  name: debug-gateway
  namespace: gloo-system
spec:
  kube: 
    envoyContainer:
      bootstrap:
        logLevel: debug
EOF
  2. Create a Gateway resource that references your custom GlooGatewayParameters.
    kubectl apply -f- <<EOF
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: debug-gateway
  namespace: gloo-system
spec:
  gatewayClassName: gloo-gateway-v2
  infrastructure:
    parametersRef:
      name: debug-gateway
      group: gloo.solo.io
      kind: GlooGatewayParameters      
  listeners:
  - protocol: HTTP
    port: 8080
    name: http
    allowedRoutes:
      namespaces:
        from: All
EOF
  3. Verify that a pod is created for your gateway proxy and that it has the pod settings that you defined in the GlooGatewayParameters resource.
    kubectl get pods -l app.kubernetes.io/name=debug-gateway -n gloo-system -o yaml
  4. Create an HTTPRoute that routes traffic to your app through the debug gateway. The following example assumes that you set up the sample httpbin app.
    kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
  namespace: httpbin
spec:
  parentRefs:
    - name: debug-gateway
      namespace: gloo-system
  hostnames:
    - "debug.com"
  rules:
    - backendRefs:
        - name: httpbin
          port: 8000
EOF
  5. Get the address of the debug gateway proxy.
    • Cloud Provider LoadBalancer
      export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-system debug-gateway -o=jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS
    • Port-forward for local testing
      kubectl port-forward deployment/debug-gateway -n gloo-system 8080:8080
  6. Send traffic through the debug gateway proxy.
    • Cloud Provider LoadBalancer
      curl -i http://$INGRESS_GW_ADDRESS:8080/headers -H "host: debug.com:8080"
    • Port-forward for local testing
      curl -i localhost:8080/headers -H "host: debug.com"
  7. Review the debug logs for the gateway proxy.
    kubectl logs -n gloo-system -l app.kubernetes.io/name=debug-gateway
    Example output:
    [2025-07-08 18:59:13.234][34][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:254] [Tags: "ConnectionId":"2"] destroying stream: 0 active remaining, readyForStream false, currentUnusedCapacity 1
[2025-07-08 18:59:14.240][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:774] [Tags: "ConnectionId":"1"] remote close
[2025-07-08 18:59:14.241][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:314] [Tags: "ConnectionId":"1"] closing socket: 0
[2025-07-08 18:59:14.243][34][debug][conn_handler] [external/envoy/source/common/listener_manager/active_stream_listener_base.cc:136] [Tags: "ConnectionId":"1"] adding to cleanup list
[2025-07-08 18:59:14.244][1][debug][main] [external/envoy/source/server/server.cc:245] flushing stats
[2025-07-08 18:59:18.232][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:774] [Tags: "ConnectionId":"2"] remote close
[2025-07-08 18:59:18.233][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:314] [Tags: "ConnectionId":"2"] closing socket: 0
[2025-07-08 18:59:18.233][34][debug][client] [external/envoy/source/common/http/codec_client.cc:107] [Tags: "ConnectionId":"2"] disconnect. resetting 0 pending requests
[2025-07-08 18:59:18.234][34][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:532] [Tags: "ConnectionId":"2"] client disconnected, failure reason: 
[2025-07-08 18:59:18.235][34][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:500] invoking 1 idle callback(s) - is_draining_for_deletion_=false

Policy not applied

As part of debugging, you might have noticed that your HTTPRoute or Gateway had an attached GlooTrafficPolicy. The GlooTrafficPolicy’s status might say Accepted and seem normal. However, when you checked the gateway configuration, the policy is not applied to the selected routes. Review the following common reasons for missing policies.
  1. Verify that the GlooTrafficPolicy is attached correctly. For example, you might use label selectors that do not match any HTTPRoute or Gateway. For more information, see Policy attachment.
  2. Confirm that you do not have multiple, conflicting policies. In general, the oldest policy is enforced. For more information, see Policy priority and merging rules.
  3. Determine if you need a Kubernetes ReferenceGrant. For example, the GlooTrafficPolicy might rely on a GatewayExtension to enable a feature such as external auth. However, the GatewayExtension might be in a different namespace than the backing external auth service.
    Example ReferenceGrant for external auth GatewayExtension:
    • The GrantExtension for external auth, HTTPRoute, and backing Service are in the app namespace, such as httpbin.
    • The external auth service is in the gloo-system namespace.
    apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: reference-grant
  namespace: gloo-system
spec:
  from:
    - group: gateway.kgateway.dev
      kind: GatewayExtension
      namespace: httpbin
  to:
    - group: ""
      kind: Service
Upgrade