The Solo Enterprise for Istio telemetry pipeline integrates with Jaeger as the tracing platform. Jaeger is an open source tool that helps you follow the path of a request as it is forwarded between microservices. The chain of events and interactions are then captured by the telemetry pipeline, and you can bring your own Jaeger instance to visualize to visualize the trace data in the Solo UI. You can use this data to troubleshoot issues in your microservices and identify bottlenecks. You can also forward the traces from the telemetry gateway to your own Jaeger tracing platform.

Request tracing is not set up in the telemetry pipeline by default. To capture traces for Istio-enabled workloads, you must instrument your workloads to generate traces and to send them to the telemetry collector agent so that they can be forwarded to the telemetry gateway and your custom Jaeger instance.

Before you begin

  1. Install either an ambient or sidecar mesh, and deploy the Bookinfo sample app to the mesh.

  2. Install the Solo Enterprise for Istio management plane.

Step 1: Enable tracing in Istio

Instrument Istio workloads to collect traces by updating your Istio installation. The steps to update Istio vary depending on how you installed Istio.

Sidecar mode

  1. Create the following gloo-extensions-config configmap. These settings enable tracing and set a sampling rate of 100% of requests. The traces are forwarded to the telemetry collector agents. For more information about the sampling rate, custom tag, and maximum path length settings, see the Istio tracing configuration docs.

      kubectl apply -n gloo-mesh -f -<<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: gloo-extensions-config
  namespace: gloo-mesh
data:
  values.istiod: |
    meshConfig:
      # Enable tracing
      enableTracing: true
      # Specify tracing settings
      defaultConfig:
        tracing:
          sampling: 100
          zipkin:
            address: gloo-telemetry-collector.gloo-mesh.svc.cluster.local:9411
EOF

  2. Restart the Istio workloads that you want to collect traces for. For example, if you deployed the Bookinfo sample app, you can restart the product page app with the following command.

      kubectl rollout restart deployment productpage-v1 -n bookinfo

  1. Get the current values for the istiod Helm release in your cluster.

      helm get values istiod -n istio-system -o yaml > istiod.yaml
open istiod.yaml

  2. Make the following edits to enable tracing and set a sampling rate of 100% of requests. The traces are forwarded to the telemetry collector agents. For more information about the sampling rate, custom tag, and maximum path length settings, see the Istio tracing configuration docs. After you make the edit, save and close the file.

      ...
meshConfig:
  # Enable tracing
  enableTracing: true
  # Specify tracing settings
  defaultConfig:
    tracing:
      sampling: 100
      zipkin:
        address: gloo-telemetry-collector.gloo-mesh.svc.cluster.local:9411

  3. Upgrade your Helm release with the updated values.

      helm upgrade istiod oci://${HELM_REPO}/istiod \
-n istio-system \
--version ${ISTIO_IMAGE} \
-f istiod.yaml

  4. Verify that the istiod pods are successfully restarted. Note that it might take a few seconds for the pods to become available.

      kubectl get pods -n istio-system | grep istiod

    Example output:

      istiod-b84c55cff-tllfr   1/1     Running   0          58s

  5. Restart the Istio workloads that you want to collect traces for. For example, if you deployed the Bookinfo sample app, you can restart the product page app with the following command.

      kubectl rollout restart deployment productpage-v1 -n bookinfo

Ambient mode

  1. Create the following gloo-extensions-config configmap. These settings enable tracing for both the ztunnel and istio components, and set a sampling rate of 100% of requests. The traces are forwarded to the telemetry collector agents. For more information about the sampling rate, custom tag, and maximum path length settings, see the Istio tracing configuration docs.

      kubectl apply -n gloo-mesh -f -<<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: gloo-extensions-config
  namespace: gloo-mesh
data:
  values.istio-ztunnel: |
    l7Telemetry:
      distributedTracing:
        # Enable distributed tracing
        enabled: true
        # OTLP endpoint to send spans to
        otlpEndpoint: "http://gloo-telemetry-collector.gloo-mesh:4317"
  values.istiod: |
    meshConfig:
      # Enable tracing
      enableTracing: true
      # Specify tracing settings
      defaultConfig:
        tracing:
          sampling: 100
          zipkin:
            address: gloo-telemetry-collector.gloo-mesh.svc.cluster.local:9411
EOF

  2. Restart the Istio workloads that you want to collect traces for. For example, if you deployed the Bookinfo sample app, you can restart the product page app with the following command.

      kubectl rollout restart deployment productpage-v1 -n bookinfo

  1. Enable tracing for the istiod component.

    1. Get the current values for the istiod Helm release in your cluster.

        helm get values istiod -n istio-system -o yaml > istiod.yaml
open istiod.yaml

    2. Make the following edits to enable tracing and set a sampling rate of 100% of requests. The traces are forwarded to the telemetry collector agents. For more information about the sampling rate, custom tag, and maximum path length settings, see the Istio tracing configuration docs. After you make the edit, save and close the file.

        ...
meshConfig:
  # Enable tracing
  enableTracing: true
  # Specify tracing settings
  defaultConfig:
    tracing:
      sampling: 100
      zipkin:
        address: gloo-telemetry-collector.gloo-mesh.svc.cluster.local:9411

    3. Upgrade your Helm release with the updated values.

        helm upgrade istiod oci://${HELM_REPO}/istiod \
-n istio-system \
--version ${ISTIO_IMAGE} \
-f istiod.yaml

    4. Verify that the istiod pods are successfully restarted. Note that it might take a few seconds for the pods to become available.

        kubectl get pods -n istio-system | grep istiod

      Example output:

        istiod-b84c55cff-tllfr   1/1     Running   0          58s

  2. Enable tracing for the ztunnel components.

    1. Get the current values for the istiod Helm release in your cluster.

        helm get values ztunnel -n istio-system -o yaml > ztunnel.yaml
open ztunnel.yaml

    2. Update the ztunnel configmap to point to the OTLP endpoint that the management plane uses for trace collection.

        ...
env:
  L7_ENABLED: true
# Add the management plane OTLP endpoint
l7Telemetry:
  distributedTracing:
    otlpEndpoint: "http://gloo-telemetry-collector.gloo-mesh:4317"

    3. Upgrade your Helm release with the updated values.

        helm upgrade ztunnel oci://${HELM_REPO}/ztunnel -n istio-system --version ${ISTIO_IMAGE} -f ztunnel.yaml

    4. Verify that the ztunnel pods are successfully restarted. Note that it might take a few seconds for the pods to become available.

        kubectl get pods -n istio-system | grep ztunnel

      Example output:

        ztunnel-tvtzn                      1/1     Running   0             40s
ztunnel-vtpjm                      1/1     Running   0             40s
ztunnel-hllxg                      1/1     Running   0             40s

  3. Optional: If you use the Istio ingress gateway with the classic Istio network API, such as by following this guide in the community Istio docs, you can enable tracing for the Istio ingress gateway. In ambient mode, ztunnel traces can be generated only for requests that are routed by Envoy through the gateway.

    1. Get the current values for the ingress gateway Helm release in your cluster.

        helm get values istio-ingressgateway -n istio-ingress -o yaml > ingress-gateway.yaml
open ingress-gateway.yaml

    2. Make the following edits to enable tracing and set a sampling rate of 100% of requests. The traces are forwarded to the telemetry collector agents. For more information about the sampling rate, custom tag, and maximum path length settings, see the Istio tracing configuration docs. After you make the edit, save and close the file.

        ...
meshConfig:
  # Enable tracing
  enableTracing: true
  # Specify tracing settings
  defaultConfig:
    tracing:
      sampling: 100
      zipkin:
        address: gloo-telemetry-collector.gloo-mesh.svc.cluster.local:9411

    3. Upgrade your Helm release with the updated values.

        helm upgrade istio-ingressgateway istio/gateway \
-n istio-ingress \
--version ${ISTIO_IMAGE} \
-f ingress-gateway.yaml

    4. Verify that the ingress gateway pods are successfully restarted and the load balancer service is assigned an external address.

        kubectl get pods,svc -n istio-ingress

      Example output:

        NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-665d46686f-nhh52   1/1     Running   0          106s
istio-ingressgateway-665d46686f-tlp5j   1/1     Running   0          2m1s
NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                                      AGE
istio-ingressgateway        LoadBalancer   10.96.252.49    <externalip>  15021:32378/TCP,80:30315/TCP,443:32186/TCP,31400:30313/TCP,15443:31632/TCP                                   2m2s

Step 2: Enable the Jaeger UI and add traces to the pipeline

Now that your traces are enabled for Istio workloads, you can configure the telemetry pipeline to collect the traces and forward the traces to your own Jaeger tracing platform.

Bring your own Jaeger

Instead of using the built-in Jaeger instance, you can configure the Gloo UI to point to your Jaeger instance instead.

  1. Install or use an existing Jaeger instance. For example, you might follow the Jaeger Operator documentation to set up Jaeger in your cluster. Do not use the jaeger.enabled setting in the Helm values file as this setting enables the built-in Jaeger instance.

  2. Get your current installation Helm values, and save them in a file.

      helm get values gloo-platform -n gloo-mesh -o yaml > gloo-single.yaml
open gloo-single.yaml

  3. In the Helm chart for the management plane, add the endpoint, port, and base path for your Jaeger instance in the glooUi section to embed the Jaeger UI into the Gloo UI, and disable the Jaeger UI that is automatically built in to Solo Enterprise for Istio. Note that your Jaeger platform must be served from a path that is different from / as this path is reserved for the built-in Jaeger instance in Solo Enterprise for Istio.

    To pick up traces from workloads and send them to your Jaeger instance, enable the built-in traces/istio pipeline and add an extra exporter.

      
jaeger: 
  enable: false
glooUi: 
  tracing: 
    endpoint: "<endpoint>"
    port: <port>
    basePath: "/<base-path>"
telemetryGateway:
  enabled: false
telemetryCollectorCustomization:
  extraExporters:
    otlp/jaeger-custom:
      endpoint: myjaeger:4317
      tls:
        insecure: true
  pipelines:
    traces/istio: 
      enabled: true
      pipeline: 
        exporters: 
        - otlp/jaeger-custom
telemetryCollector:
  enabled: true

  4. Upgrade your installation by using your updated values file.

      helm upgrade gloo-platform gloo-platform/gloo-platform \
   --namespace gloo-mesh \
   --version ${MGMT_VERSION} \
   --values gloo-single.yaml

  5. Verify that your custom Jaeger settings were added to the telemetry collector configmap.

      kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml

  6. Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pod.

      kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent

  1. Get the Helm values files for your current version.

    1. Get your current values for the management plane.
        helm get values gloo-platform -n gloo-mesh -o yaml --kube-context ${context1} > mgmt-plane.yaml
open mgmt-plane.yaml
    2. Get your current values for the data plane.
        helm get values gloo-platform -n gloo-mesh -o yaml --kube-context ${context2} > data-plane.yaml
open data-plane.yaml

  2. In the Helm chart for the management plane, add the endpoint, port, and base path for your Jaeger instance in the glooUi section to embed the Jaeger UI into the Gloo UI and disable the Jaeger UI that is automatically built in to Solo Enterprise for Istio. Note that your Jaeger platform must be served from a path that is different from / as this path is reserved for the built-in Jaeger instance in Solo Enterprise for Istio.

    To forward traces from the telemetry gateway to your Jaeger instance, set up an extra exporter on the telemetry gateway with the details of your Jaeger endpoint, and add that exporter to the default traces/jaeger pipeline in your Helm values file. For more information about how to set up the exporter, see the OTLP gRPC exporter documentation.

    To collect traces from workloads in the management cluster, enable the traces/istio pipeline in the telemetry collector agent.

      
jaeger: 
  enable: false
glooUi: 
  tracing: 
    endpoint: "<endpoint>"
    port: <port>
    basePath: "/<base-path>"
telemetryGateway:
  enabled: true
  service:
    type: ClusterIP
telemetryGatewayCustomization:
  extraExporters:
    otlp/jaeger-custom:
      endpoint: myjaeger:4317
      tls:
        insecure: true
  pipelines:
    traces/jaeger: 
      enabled: true
      pipeline: 
        exporters: 
        - otlp/jaeger-custom
telemetryCollector:
  enabled: true
telemetryCollectorCustomization:
  pipelines:
    traces/istio:
      enabled: true
  extraExporters:
    otlp/jaeger:
      endpoint: gloo-jaeger-collector.gloo-mesh.svc:4317
      tls:
        insecure: true

  3. In the Helm values file for the data plane, enable the default traces/istio pipeline in your Helm values file to collect traces from Istio-enabled workloads. The traces/istio pipeline is configured on the collector agents and sends traces to the telemetry gateway in the cluster where the management plane is deployed.

      
telemetryCollector:
  config:
    exporters:
      otlp:
        endpoint: gloo-telemetry-gateway.gloo-mesh:4317
telemetryCollectorCustomization:
  pipelines:
    traces/istio:
      enabled: true

  4. Upgrade the management plane release.

      helm upgrade gloo-platform gloo-platform/gloo-platform \
  --kube-context ${context1} \
  --namespace gloo-mesh \
  -f mgmt-plane.yaml \
  --version ${MGMT_VERSION}

  5. Verify that your settings are applied in the management cluster.

    1. Verify that the tracing settings were added to the telemetry collector and gateway configmaps.

        kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context ${context1}
kubectl get configmap gloo-telemetry-gateway-config -n gloo-mesh -o yaml --context ${context1}

    2. Perform a rollout restart of the telemetry gateway deployment and the telemetry collector daemon set to force your configmap changes to be applied to the telemetry gateway and collector agent pods.

        kubectl rollout restart -n gloo-mesh deployment/gloo-telemetry-gateway --context ${context1}
kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context ${context1}

  6. Upgrade the data plane release.

      helm upgrade gloo-platform gloo-platform/gloo-platform \
  --kube-context ${context2} \
  --namespace gloo-mesh \
  -f data-plane.yaml \
  --version ${MGMT_VERSION}

  7. Verify that your settings are applied in the workload cluster.

    1. Verify that the tracing settings were added to the telemetry collector configmap.

        kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context ${context2}

    2. Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pods.

        kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context ${context2}

Deprecated: Use the built-in Jaeger

During your Solo Enterprise for Istio installation, you can enable Jaeger as the tracing platform for your environment and embed the Jaeger UI in to the Gloo UI.

  1. Get your current installation Helm values, and save them in a file.

      helm get values gloo-platform -n gloo-mesh -o yaml > gloo-single.yaml
open gloo-single.yaml

  2. Add the following settings to enable Jaeger and the built-in traces/istio pipeline. The traces/istio pipeline is set up on the collector agent and collects traces from Istio-enabled workloads. Traces are sent to the built-in Jaeger tracing platform.

      
jaeger:
  enabled: true
telemetryCollectorCustomization:
  pipelines:
    traces/istio:
      enabled: true

  3. Upgrade your installation by using your updated values file.

      helm upgrade gloo-platform gloo-platform/gloo-platform \
   --namespace gloo-mesh \
   --version ${MGMT_VERSION} \
   --values gloo-single.yaml

  4. Verify that you see a gloo-jaeger deployment in your cluster.

      kubectl get deployment gloo-jaeger -n gloo-mesh

  5. Verify that your custom Jaeger settings were added to the telemetry collector configmap.

      kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml

  6. Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pod.

      kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent

  1. Get your current Helm values file for the management plane.

      helm get values gloo-platform -n gloo-mesh -o yaml --kube-context ${context1} > mgmt-plane.yaml
open mgmt-plane.yaml

  2. Enable the built-in Jaeger platform and the traces/jaeger pipeline to forward traces from the telemetry gateway to the built-in Jaeger platform. Because a telemetry collector agent is also deployed to the cluster where the management plane is deployed, you must enable the traces/istio pipeline on the collector for traces to be collected in that cluster.

      
jaeger:
  enabled: true
telemetryGateway:
  enabled: true
telemetryGatewayCustomization:
  pipelines:
    traces/jaeger:
      enabled: true
telemetryCollector:
  enabled: true
telemetryCollectorCustomization:
  pipelines:
    traces/istio:
      enabled: true

  3. Upgrade the management plane release.

      helm upgrade gloo-platform gloo-platform/gloo-platform \
  --kube-context ${context1} \
  --namespace gloo-mesh \
  -f mgmt-plane.yaml \
  --version ${MGMT_VERSION}

  4. Verify that your settings are applied in the management cluster.

    1. Verify that you see a gloo-jaeger deployment in your cluster.

        kubectl get deployment gloo-jaeger -n gloo-mesh --context ${context1}

    2. Verify that the tracing settings were added to the telemetry collector and gateway configmaps.

        kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context ${context1}
kubectl get configmap gloo-telemetry-gateway-config -n gloo-mesh -o yaml --context ${context1}

    3. Perform a rollout restart of the telemetry gateway deployment and the telemetry collector daemon set to force your configmap changes to be applied to the telemetry gateway and collector agent pods.

        kubectl rollout restart -n gloo-mesh deployment/gloo-telemetry-gateway --context ${context1}
kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context ${context1}

  5. Get your current Helm values file for the data plane.

      helm get values gloo-platform -n gloo-mesh -o yaml --kube-context ${context2} > data-plane.yaml
open data-plane.yaml

  6. Enable the traces/istio pipeline to pick up traces from Istio-enabled workloads and forward them to the telemetry gateway.

      
telemetryCollector:
  enabled: true
telemetryCollectorCustomization:
  pipelines:
    traces/istio:
      enabled: true

  7. Upgrade the data plane release.

      helm upgrade gloo-platform gloo-platform/gloo-platform \
  --kube-context ${context2} \
  --namespace gloo-mesh \
  -f data-plane.yaml \
  --version ${MGMT_VERSION}

  8. Verify that your settings are applied in the workload cluster.

    1. Verify that the tracing settings were added to the telemetry collector configmap.

        kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context ${context2}

    2. Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pods.

        kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context ${context2}

Step 3: Verify tracing

Open the Gloo UI and verify that traces are collected for your Istio workloads.

Sidecar mode

  1. Open the Gloo UI. The Gloo UI is served from the gloo-mesh-ui service on port 8090. In multicluster setups, the UI is installed in the cluster where the management plane is deployed. You can connect by using the meshctl or kubectl CLIs.

    • meshctl: For more information, see the CLI documentation.
        meshctl dashboard
    • kubectl:
      1. Port-forward the gloo-mesh-ui service on 8090.
          kubectl port-forward -n gloo-mesh svc/gloo-mesh-ui 8090:8090
      2. Open your browser and connect to http://localhost:8090.

  2. From the menu, select Tracing and verify that the Jaeger UI opens.

    Figure: Jaeger UI
    Figure: Jaeger UI
    Figure: Jaeger UI
    Figure: Jaeger UI

  3. Send a few sample requests to your Istio workloads. Each request produces Istio traces that are sent to the Jaeger instance that you configured. For example, if you deployed the Bookinfo sample app, use the following steps to produce traces.

    1. Port-forward the product page app.
        kubectl port-forward deployment/productpage-v1 -n bookinfo 9080
    2. Open the product page app.
    3. Refresh the page multiple times.

  4. Wait a few seconds and verify that traces are displayed in the Gloo UI.

    Figure: Product page traces
    Figure: Product page traces
    Figure: Product page traces
    Figure: Product page traces

Ambient mode

  1. Open the Gloo UI. The Gloo UI is served from the gloo-mesh-ui service on port 8090. In multicluster setups, the UI is installed in the cluster where the management plane is deployed. You can connect by using the meshctl or kubectl CLIs.

    • meshctl: For more information, see the CLI documentation.
        meshctl dashboard
    • kubectl:
      1. Port-forward the gloo-mesh-ui service on 8090.
          kubectl port-forward -n gloo-mesh svc/gloo-mesh-ui 8090:8090
      2. Open your browser and connect to http://localhost:8090.

  2. From the menu, select Tracing and verify that the Jaeger UI opens.

    Figure: Jaeger UI
    Figure: Jaeger UI
    Figure: Jaeger UI
    Figure: Jaeger UI

  3. Send a few sample requests to your Istio workloads. Each request produces Istio traces that are sent to the Jaeger instance that you configured. For example, if you deployed the Bookinfo sample app, use the following steps to produce traces.

    1. If you have not already, expose Bookinfo externally.
        kubectl -n bookinfo apply -f https://raw.githubusercontent.com/istio/istio/1.29.1/samples/bookinfo/networking/bookinfo-gateway.yaml --context ${context2}
    2. Save the external address of your ingress gateway in an environment variable.
        export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-mesh-gateways --context ${context2} istio-ingressgateway -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS
    3. Open the product page UI and refresh the page multiple times.
        open http://$INGRESS_GW_ADDRESS/productpage

  4. Wait a few seconds and verify that traces are displayed in the Gloo UI. For example, you can check traces through individual ztunnel pods to see traffic requests to Bookinfo services.

    Figure: Product page traces
    Figure: Product page traces