Distributed tracing helps you track requests across multiple services in your distributed service mesh. Sampling the requests in your mesh can give you insight into request latency, serialization, and parallelism. For more information, see the Istio tracing documentation.

To get started, enable the default Envoy distributed tracing in your Istio installation to trace requests across your service mesh, analyze service dependencies and find bottlenecks or high latency services in your mesh. Then, use a tracing provider, such as Jaeger, to collect and visualize the sampled spans.

Before you begin

  1. Complete the multicluster getting started guide to set up the following testing environment.

    • Three clusters along with environment variables for the clusters and their Kubernetes contexts.
    • The Gloo meshctl CLI, along with other CLI tools such as kubectl and istioctl.
    • The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
    • Istio installed in the workload clusters.
    • A simple Gloo workspace setup.
  2. Install Bookinfo and other sample apps.

Enable tracing in Istio

  1. Instrument Istio workloads to collect traces by updating your Istio installation. The steps to update Istio vary depending on how you installed Istio.

    1. Edit the IstioLifecycleManager resource in your cluster.

        kubectl edit istiolifecyclemanager istiod-control-plane -n gloo-mesh --context $MGMT_CONTEXT

    2. Add the following configuration to the istioOperatorSpec.meshConfig section to set a sampling rate of 100% of requests. The traces are forwarded to the Gloo telemetry collector agents. For more information about the sampling rate, custom tag, and maximum path length settings, see the Istio tracing configuration docs.

        
meshConfig:      
  enableTracing: true
  defaultConfig: 
    tracing: 
      sampling: 100
      zipkin:
        address: gloo-telemetry-collector.gloo-mesh.svc.cluster.local:9411

    3. Verify that your configuration is applied in the cluster.

        export REVISION=$(kubectl get pod -L app=istiod -n istio-system -o jsonpath='{.items[0].metadata.labels.istio\.io/rev}')
kubectl get istiooperator istiod-control-plane --context $REMOTE_CONTEXT -n gm-iop-$REVISION -o yaml

    Refer to the Istio documentation for instructions on how to update your Istio installation. The following example meshConfig options set a sampling rate of 100% of requests. These traces are sent to the Gloo telemetry collector agent.

      
meshConfig:      
  enableTracing: true
  defaultConfig: 
    tracing: 
      sampling: 100
      zipkin:
        address: gloo-telemetry-collector.gloo-mesh.svc.cluster.local:9411

    For more information about the sampling rate, custom tag, and maximum path length settings, see the Istio tracing configuration docs.

  2. Restart the Istio workloads that you want to collect traces for. For example, if you deployed the Bookinfo sample app as part of the Get started guide, you can restart the product page app with the following command.

      kubectl rollout restart deployment productpage-v1 -n bookinfo --context $REMOTE_CONTEXT

  3. Begin tracing the requests in your service mesh and view the sampled spans.

For more information about setting a tracer, see the Istio meshConfig.defaultConfig.tracing reference docs.