Skip to content
home
2.1.x
Operations
You are viewing the latest documentation for Solo Enterprise for kgateway, formerly known as Gloo Gateway. To access the documentation for older Gloo Gateway versions, such as 2.0 and 1.x, use the version switcher.

Debug

Page as Markdown
Debug your gateway setup.
Use built-in tools to troubleshoot issues in your Solo Enterprise for kgateway setup.
Solo Enterprise for kgateway consists of the control plane and an Envoy-based kgateway data plane. If you experience issues in your environment, such as policies that are not applied or traffic that is not routed correctly, in a lot of cases, these errors can be observed at the proxy.
Debug the control plane

  1. Enable port-forwarding on the control plane.
    kubectl port-forward deploy/enterprise-kgateway -n kgateway-system 9095
  2. In your browser, open the admin server debugging interface: http://localhost:9095/.
    Figure: Admin server debugging interface.
    Figure: Admin server debugging interface.
  3. Select one of the endpoints to continue debugging. Review the following table to understand this configuration.
    EndpointDescription
    /debug/pprofView the pprof profile of the control plane. A profile shows you the stack traces of the call sequences, such as Go routines, that led to particular events, such as memory allocation. The endpoint includes descriptions of each available profile.
    /loggingReview the current logging levels of each component in the control plane. You can also interactively set the log level by component, such as to enable DEBUG logs.
    /snapshots/krtView the current krt snapshot, or the point-in-time view of the transformed Kubernetes resources and their sync status that the control plane processed. These resources are then used to generate gateway configuration that is sent to the gateway proxies for routing decisions.
    /snapshots/xdsView the current xDS snapshot, or the Envoy-specific configuration (such as Listeners, Routes, Backends, and Workloads) that is being sent to and applied by Envoy gateway proxies. These snapshots show the final translated configuration that Envoy gateway proxies use for routing decisions. For snapshots to be present, you must have a Gateway resource that creates a connected, Envoy-based kgateway proxy.
Debug your gateway setup

  1. Make sure that the Solo Enterprise for kgateway control plane and gateway proxies are running. For any pod that is not running, describe the pod for more details.
    kubectl get pods -n kgateway-system
  2. Check the HTTPRoutes for the status of the route and any attached policies.
    kubectl get httproutes -A
    kubectl get httproute <name> -n <namespace> -o yaml
  3. Access the debugging interface of your gateway proxy on your localhost. Configuration might be missing on the gateway or might be applied to the wrong route. For example, if you apply multiple policies to the same route by using the targetRefs section, only the oldest policy is applied. The newer policy configuration might be ignored and not applied to the gateway.
kubectl port-forward deploy/http -n kgateway-system 19000 &  
Figure: Debugging interface of the gateway proxy.
Figure: Debugging interface of the gateway proxy.
Review the following table for common endpoints that can help troubleshoot your setup further.
EndpointDescription
config_dumpGet the configuration that is available in the Envoy proxy. Any kgateway resources that you create are translated in to Envoy configuration. Depending on whether or not you enabled resource validation, you might have applied invalid configuration that is rejected Envoy. You can also use glooctl proxy dump to get the Envoy proxy configuration.
listenersSee the listeners that are configured on your gateway.
loggingReview the log level that is set for each component.
stats/prometheusView metrics that Envoy emitted and sent to the built-in Prometheus instance.
  1. Review the logs for each component. Each component logs the sync loops that it runs, such as syncing with various environment signals like the Kubernetes API.
    You can fetch the latest logs for all the components with the following command.
# Solo Enterprise for kgateway control plane
kubectl logs -n kgateway-system deployment/enterprise-kgateway

# Replace $GATEWAY_NAME with the name of your gateway.
export GATEWAY_NAME=http
kubectl logs -n kgateway-system deployment/$GATEWAY_NAME
Set gateway proxy debug logging

You can set the log level for the Envoy proxy to get more detailed logs. Envoy log level options include trace, debug, info, warn, error, critical, and off. The default log level is info. For more information, see Debugging Envoy.
  1. Create a EnterpriseKgatewayParameters resource to add any custom settings to the gateway. For other settings, see the EnterpriseKgatewayParameters API docs or check out the Gateway customization guides.
    kubectl apply -f- <<EOF
apiVersion: enterprisekgateway.solo.io/v1alpha1
kind: EnterpriseKgatewayParameters
metadata:
  name: debug-gateway
  namespace: kgateway-system
spec:
  kube: 
    envoyContainer:
      bootstrap:
        logLevel: debug
EOF
  2. Create a Gateway resource that references your custom EnterpriseKgatewayParameters.
    kubectl apply -f- <<EOF
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: debug-gateway
  namespace: kgateway-system
spec:
  gatewayClassName: enterprise-kgateway
  infrastructure:
    parametersRef:
      name: debug-gateway
      group: enterprisekgateway.solo.io
      kind: EnterpriseKgatewayParameters      
  listeners:
  - protocol: HTTP
    port: 8080
    name: http
    allowedRoutes:
      namespaces:
        from: All
EOF
  3. Verify that a pod is created for your gateway proxy and that it has the pod settings that you defined in the EnterpriseKgatewayParameters resource.
    kubectl get pods -l app.kubernetes.io/name=debug-gateway -n kgateway-system -o yaml
  4. Create an HTTPRoute that routes traffic to your app through the debug gateway. The following example assumes that you set up the sample httpbin app.
    kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
  namespace: httpbin
spec:
  parentRefs:
    - name: debug-gateway
      namespace: kgateway-system
  hostnames:
    - "debug.com"
  rules:
    - backendRefs:
        - name: httpbin
          port: 8000
EOF
  5. Get the address of the debug gateway proxy.
    • Cloud Provider LoadBalancer
      export INGRESS_GW_ADDRESS=$(kubectl get svc -n kgateway-system debug-gateway -o=jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS
    • Port-forward for local testing
      kubectl port-forward deployment/debug-gateway -n kgateway-system 8080:8080
  6. Send traffic through the debug gateway proxy.
    • Cloud Provider LoadBalancer
      curl -i http://$INGRESS_GW_ADDRESS:8080/headers -H "host: debug.com:8080"
    • Port-forward for local testing
      curl -i localhost:8080/headers -H "host: debug.com"
  7. Review the debug logs for the gateway proxy.
    kubectl logs -n kgateway-system -l app.kubernetes.io/name=debug-gateway
    Example output:
    [2025-07-08 18:59:13.234][34][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:254] [Tags: "ConnectionId":"2"] destroying stream: 0 active remaining, readyForStream false, currentUnusedCapacity 1
[2025-07-08 18:59:14.240][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:774] [Tags: "ConnectionId":"1"] remote close
[2025-07-08 18:59:14.241][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:314] [Tags: "ConnectionId":"1"] closing socket: 0
[2025-07-08 18:59:14.243][34][debug][conn_handler] [external/envoy/source/common/listener_manager/active_stream_listener_base.cc:136] [Tags: "ConnectionId":"1"] adding to cleanup list
[2025-07-08 18:59:14.244][1][debug][main] [external/envoy/source/server/server.cc:245] flushing stats
[2025-07-08 18:59:18.232][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:774] [Tags: "ConnectionId":"2"] remote close
[2025-07-08 18:59:18.233][34][debug][connection] [external/envoy/source/common/network/connection_impl.cc:314] [Tags: "ConnectionId":"2"] closing socket: 0
[2025-07-08 18:59:18.233][34][debug][client] [external/envoy/source/common/http/codec_client.cc:107] [Tags: "ConnectionId":"2"] disconnect. resetting 0 pending requests
[2025-07-08 18:59:18.234][34][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:532] [Tags: "ConnectionId":"2"] client disconnected, failure reason: 
[2025-07-08 18:59:18.235][34][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:500] invoking 1 idle callback(s) - is_draining_for_deletion_=false

Policy not applied

As part of debugging, you might have noticed that your HTTPRoute or Gateway had an attached EnterpriseKgatewayTrafficPolicy. The EnterpriseKgatewayTrafficPolicy’s status might say Accepted and seem normal. However, when you checked the gateway configuration, the policy is not applied to the selected routes. Review the following common reasons for missing policies.
  1. Verify that the EnterpriseKgatewayTrafficPolicy is attached correctly. For example, you might use label selectors that do not match any HTTPRoute or Gateway. For more information, see Policy attachment.
  2. Confirm that you do not have multiple, conflicting policies. In general, the oldest policy is enforced. For more information, see Policy priority and merging rules.
  3. Determine if you need a Kubernetes ReferenceGrant. For example, the EnterpriseKgatewayTrafficPolicy might rely on a GatewayExtension to enable a feature such as external auth. However, the GatewayExtension might be in a different namespace than the backing external auth service.
    Example ReferenceGrant for external auth GatewayExtension:
    • The GrantExtension for external auth, HTTPRoute, and backing Service are in the app namespace, such as httpbin.
    • The external auth service is in the kgateway-system namespace.
    apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: reference-grant
  namespace: kgateway-system
spec:
  from:
    - group: gateway.kgateway.dev
      kind: GatewayExtension
      namespace: httpbin
  to:
    - group: ""
      kind: Service
Upgrade