Request tracing

Sample request traces to monitor the traffic and health of your ingress gateway.

Distributed tracing helps you track requests that enter your cluster through the ingress gateway. Sampling requests can give you insight into request latency, serialization, and parallelism. For more information, see the Istio tracing documentation.

To get started, enable the default Envoy distributed tracing in your Istio installation, which allows you to trace requests that come in to your ingress gateway, analyze service dependencies, and find bottlenecks or high latency services in your cluster. Then, use a tracing provider, such as the built-in Jaeger platform, to collect and visualize the sampled spans.

1. Enable tracing in the Gloo OTel pipeline.

   This guide walks you through how to set up tracing with the built-in Jaeger platform, which can be used for testing or demo purposes. However, if you plan to set up tracing for your production environment, create your own custom tracing pipeline in the Gloo telemetry pipeline to forward traces to a tracing platform that is managed by your organization and hardened for production. Alternatively, you can send production traces to a SaaS backend. Use the steps in Bring your own Jaeger instance as a guide to integrate your own tracing platform.

2. Instrument the ingress gateway to produce traces by updating the Istio control plane.

   1. Edit the IstioLifecycleManager resource in your cluster.

      kubectl edit istiolifecyclemanager gloo-platform -n gloo-mesh
      

   2. Add the following configuration to the meshConfig section.

      ...
      meshConfig:      
        enableTracing: true
        defaultConfig: 
          tracing: 
            sampling: 100
            zipkin:
              address: gloo-telemetry-collector.gloo-mesh.svc.cluster.local:9411
      ...
      

3. Verify that your configuration is applied in the cluster.

   export REVISION=$(kubectl get pod -L app=istiod -n istio-system -o jsonpath='{.items[0].metadata.labels.istio\.io/rev}')
   kubectl get istiooperator istio-ingressgateway-$REVISION -n gm-iop-$REVISION -o yaml
   

4. Restart the ingress gateway to start collecting traces.

   kubectl rollout restart deployment istio-ingressgateway-$REVISION -n gloo-mesh-gateways  
   

5. Send a few sample requests to the product page app. Each request produces Istio traces that are sent to the Jaeger instance that you configured.

   export INGRESS_GW_IP=$(kubectl get svc -n gloo-mesh-gateways istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   curl -vik http://www.example.com:80/productpage --resolve www.example.com:80:$INGRESS_GW_IP
   

6. Open the Gloo UI and go to the Tracing menu to view your traces.

   

For more information about the sampling rate, custom tag, and maximum path length settings, see the Istio tracing configuration docs.