Request traces

Configure Istio workloads to send request traces to your own Jaeger instance.

Jaeger is an open source distributed tracing platform that helps you follow the path of a request as it travels through your microservices. Configure Istio workloads to generate traces and send them to your own Jaeger instance. Traces are not currently displayed in the Solo UI, but can be viewed directly in the Jaeger UI.

Before you begin

Install an ambient mesh and deploy the Bookinfo sample app. For example, you can follow the get started guide to install an ambient mesh and Bookinfo.
Install the Solo UI.

Step 1: Deploy Jaeger

Deploy a Jaeger instance in your cluster. For testing purposes, you can use the Jaeger all-in-one deployment. For production use, follow the Jaeger on Kubernetes documentation to deploy a hardened Jaeger instance.
Save the namespace where you deploy Jaeger in an environment variable.
```
export JAEGER_NAMESPACE=<jaeger-namespace>
```

Step 2: Enable tracing in Istio

Instrument Istio workloads to collect traces by updating your Istio installation. The steps to update Istio vary depending on how you installed Istio.

Enable tracing for the istiod component.

Get the current values for the istiod Helm release in your cluster.

helm get values istiod -n istio-system -o yaml > istiod.yaml
open istiod.yaml

Add the following settings to enable tracing and set a sampling rate of 100% of requests. Save and close the file when done.

meshConfig:
  enableTracing: true
  defaultConfig:
    tracing:
      sampling: 100
      zipkin:
        address: jaeger-collector.${JAEGER_NAMESPACE}.svc.cluster.local:9411

Upgrade your Helm release with the updated values.

helm upgrade istiod oci://${HELM_REPO}/istiod \
-n istio-system \
--version ${ISTIO_IMAGE} \
-f istiod.yaml

Verify that the istiod pods are successfully restarted.

kubectl get pods -n istio-system | grep istiod

Example output:

istiod-b84c55cff-tllfr   1/1     Running   0          58s

Enable tracing for the ztunnel component.

Get the current values for the ztunnel Helm release in your cluster.

helm get values ztunnel -n istio-system -o yaml > ztunnel.yaml
open ztunnel.yaml

Add the following settings to send ztunnel traces to your Jaeger collector via OTLP. Save and close the file when done.

env:
  L7_ENABLED: true
l7Telemetry:
  distributedTracing:
    otlpEndpoint: "http://jaeger-collector.${JAEGER_NAMESPACE}:4317"

Upgrade your Helm release with the updated values.

helm upgrade ztunnel oci://${HELM_REPO}/ztunnel -n istio-system --version ${ISTIO_IMAGE} -f ztunnel.yaml

Verify that the ztunnel pods are successfully restarted.

kubectl get pods -n istio-system | grep ztunnel

Example output:

ztunnel-tvtzn   1/1     Running   0   40s
ztunnel-vtpjm   1/1     Running   0   40s
ztunnel-hllxg   1/1     Running   0   40s

Optional: If you use the Istio ingress gateway with the classic Istio network API, you can enable tracing for the ingress gateway. In ambient mode, ztunnel traces can be generated only for requests that are routed by Envoy through the gateway.

Get the current values for the ingress gateway Helm release in your cluster.

helm get values istio-ingressgateway -n istio-ingress -o yaml > ingress-gateway.yaml
open ingress-gateway.yaml

Add the following settings. Save and close the file when done.

meshConfig:
  enableTracing: true
  defaultConfig:
    tracing:
      sampling: 100
      zipkin:
        address: jaeger-collector.${JAEGER_NAMESPACE}.svc.cluster.local:9411

Upgrade your Helm release with the updated values.

helm upgrade istio-ingressgateway istio/gateway \
-n istio-ingress \
--version ${ISTIO_IMAGE} \
-f ingress-gateway.yaml

Create the following gloo-extensions-config configmap. These settings enable tracing for both the ztunnel and istiod components, and set a sampling rate of 100% of requests. Traces are sent directly to your Jaeger collector. For more information about the sampling rate, custom tag, and maximum path length settings, see the Istio tracing configuration docs.

kubectl apply -n gloo-mesh -f -<<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: gloo-extensions-config
  namespace: gloo-mesh
data:
  values.istio-ztunnel: |
    l7Telemetry:
      distributedTracing:
        enabled: true
        otlpEndpoint: "http://jaeger-collector.${JAEGER_NAMESPACE}:4317"
  values.istiod: |
    meshConfig:
      enableTracing: true
      defaultConfig:
        tracing:
          sampling: 100
          zipkin:
            address: jaeger-collector.${JAEGER_NAMESPACE}.svc.cluster.local:9411
EOF

Restart the Istio workloads that you want to collect traces for. For example, if you deployed the Bookinfo sample app:
```
kubectl rollout restart deployment productpage-v1 -n bookinfo
```

Step 3: Verify traces in Jaeger

Port-forward the Jaeger UI.

kubectl port-forward -n ${JAEGER_NAMESPACE} svc/jaeger-query 16686:16686

Send requests to your Istio workloads to generate traces. For example, if you deployed the Bookinfo sample app:
1. Port-forward the product page app.
  kubectl port-forward deployment/productpage-v1 -n bookinfo 9080
2. Open the product page app and refresh the page multiple times.
Open the Jaeger UI in your browser, select a service from the Service dropdown, and click Find Traces to view the generated traces.

Access logs ClickHouse data store