Add Istio request traces
The Gloo telemetry pipeline integrates with Jaeger as the tracing platform. Jaeger is an open source tool that helps you follow the path of a request as it is forwarded between microservices. The chain of events and interactions are then captured by the Gloo telemetry pipeline and visualized in the Jaeger UI that is embedded in the Gloo UI. You can use this data to troubleshoot issues in your microservices and identify bottlenecks. You can also forward the traces from the Gloo telemetry gateway to your own Jaeger tracing platform.
This guide walks you through how to set up tracing with either the built-in or your own Jaeger platform. Use the built-in Jaeger instance testing or demo purposes. If you plan to set up tracing for your production environment, create your own custom tracing pipeline in the Gloo telemetry pipeline to forward traces to a tracing platform that is managed by your organization and hardened for production. Alternatively, you can send production traces to a SaaS backend. Use the steps in Bring your own Jaeger instance as a guide to integrate your own tracing platform.
Before you begin
Follow the getting started instructions to:
- Set up Gloo Gateway in a single cluster.
- Deploy sample apps.
- Configure an HTTP listener on your gateway and set up basic routing for the sample apps.
Step 1: Enable tracing in Istio
Instrument Istio workloads to collect traces by updating your Istio installation. The steps to update Istio vary depending on how you installed Istio.
-
Edit the IstioLifecycleManager resource in your cluster.
kubectl edit istiolifecyclemanager istiod-control-plane -n gloo-mesh
-
Add the following configuration to the
meshConfig
section to set a sampling rate of 100% of requests. The traces are forwarded to the Gloo telemetry collector agents. For more information about the sampling rate, custom tag, and maximum path length settings, see the Istio tracing configuration docs.meshConfig: enableTracing: true defaultConfig: tracing: sampling: 100 zipkin: address: gloo-telemetry-collector.gloo-mesh.svc.cluster.local:9411
-
Verify that your configuration is applied in the cluster.
export REVISION=$(kubectl get pod -L app=istiod -n istio-system -o jsonpath='{.items[0].metadata.labels.istio\.io/rev}') kubectl get istiooperator istio-ingressgateway-$REVISION -n gm-iop-$REVISION -o yaml
-
Restart the Istio workloads that you want to collect traces for. For example if you deployed the Bookinfo sample app as part of the Get started guide, you can restart the product page app with the following command.
kubectl rollout restart deployment productpage-v1 -n bookinfo
-
Refer to the Istio documentation for instructions on how to update your Istio installation. The following example
meshConfig
options set a sampling rate of 100% of requests. These traces are sent to the Gloo telemetry collector agent.meshConfig: enableTracing: true defaultConfig: tracing: sampling: 100 zipkin: address: gloo-telemetry-collector.gloo-mesh.svc.cluster.local:9411
For more information about the sampling rate, custom tag, and maximum path length settings, see the Istio tracing configuration docs.
-
Restart the Istio workloads that you want to collect traces for. For example if you deployed the Bookinfo sample app as part of the Get started guide, you can restart the product page app with the following command.
kubectl rollout restart deployment productpage-v1 -n bookinfo
Step 2: Enable the Jaeger UI and add traces to the pipeline
Now that you enabled traces for Istio workloads, you can configure the Gloo telemetry pipeline. Choose from the following options:
- Send traces to the default Jaeger instance that is built in to the Gloo UI.
- Forward traces to your own Jaeger tracing platform.
Use the built-in Jaeger
During your Gloo Gateway installation, you can enable Jaeger as the tracing platform for your Gloo environment and embed the Jaeger UI in to the Gloo UI.
-
Get your current installation Helm values, and save them in a file.
helm get values gloo-platform -n gloo-mesh -o yaml > gloo-gateway-single.yaml open gloo-gateway-single.yaml
-
Enable Jaeger in your environment by adding the following configuration to your Helm values file.
jaeger: enabled: true
-
Enable the built-in
traces/istio
pipeline. Thetraces/istio
pipeline is set up on the Gloo collector agent and collects traces from Istio-enabled workloads. Traces are sent to the built-in Jaeger tracing platform.telemetryCollector: enabled: true config: exporters: otlp: endpoint: gloo-telemetry-gateway.gloo-mesh:4317 telemetryCollectorCustomization: pipelines: traces/istio: enabled: true telemetryGateway: enabled: false
-
Upgrade your installation by using your updated values file.
helm upgrade gloo-platform gloo-platform/gloo-platform \ --namespace gloo-mesh \ -f gloo-gateway-single.yaml \ --version $UPGRADE_VERSION
-
Verify that you see a
gloo-jaeger
deployment in your cluster.kubectl get deployment gloo-jaeger -n gloo-mesh
-
Verify that your custom Jaeger settings were added to the Gloo telemetry collector configmap.
kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml
-
Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pod.
kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent
-
Get the Helm values files for your current version.
- Get your current values for the management cluster.
helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $MGMT_CONTEXT > mgmt-server.yaml open mgmt-server.yaml
- Get your current values for the workload clusters.
helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $REMOTE_CONTEXT > agent.yaml open agent.yaml
- Get your current values for the management cluster.
-
In the Helm values file for the management cluster, enable the built-in Jaeger platform and the
traces/jaeger
pipeline to forward traces from the telemetry gateway to the built-in Jaeger platform. Because a Gloo telemetry collector agent is also deployed to the management cluster, you must enable thetraces/istio
pipeline on the collector for traces to be collected in the management cluster.jaeger: enabled: true telemetryGateway: enabled: true telemetryGatewayCustomization: pipelines: traces/jaeger: enabled: true telemetryCollector: enabled: true telemetryCollectorCustomization: pipelines: traces/istio: enabled: true
-
In the Helm values file for the workload cluster, enable the
traces/istio
pipeline to pick up traces from Istio-enabled workloads and forward them to the Gloo telemetry gateway.telemetryCollector: enabled: true telemetryCollectorCustomization: pipelines: traces/istio: enabled: true
-
Upgrade the management cluster.
helm upgrade gloo-platform gloo-platform/gloo-platform \ --kube-context $MGMT_CONTEXT \ --namespace gloo-mesh \ -f mgmt-server.yaml \ --version $UPGRADE_VERSION
-
Verify that your settings are applied in the management cluster.
-
Verify that you see a
gloo-jaeger
deployment in your cluster.kubectl get deployment gloo-jaeger -n gloo-mesh --context $MGMT_CONTEXT
-
Verify that the tracing settings were added to the Gloo telemetry collector and gateway configmaps.
kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context $MGMT_CONTEXT kubectl get configmap gloo-telemetry-gateway-config -n gloo-mesh -o yaml --context $MGMT_CONTEXT
-
Perform a rollout restart of the telemetry gateway deployment and the telemetry collector daemon set to force your configmap changes to be applied to the telemetry gateway and collector agent pods.
kubectl rollout restart -n gloo-mesh deployment/gloo-telemetry-gateway --context $MGMT_CONTEXT kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context $MGMT_CONTEXT
-
-
Upgrade the workload cluster.
helm upgrade gloo-platform gloo-platform/gloo-platform \ --kube-context $REMOTE_CONTEXT \ --namespace gloo-mesh \ -f agent.yaml \ --version $UPGRADE_VERSION
-
Verify that your settings are applied in the workload cluster.
-
Verify that the tracing settings were added to the Gloo telemetry collector configmap.
kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context $REMOTE_CONTEXT
-
Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pods.
kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context $REMOTE_CONTEXT
-
Bring your own Jaeger
Instead of using the built-in Jaeger instance, you can configure the Gloo UI to point to your Jaeger instance instead.
-
Install or use an existing Jaeger instance. For example, you might follow the Jaeger documentation to set up Jaeger in your cluster. Do not use the
jaeger.enabled
setting in the Helm values file as this setting enables the built-in Jaeger instance. -
Get your current installation Helm values, and save them in a file.
helm get values gloo-platform -n gloo-mesh -o yaml > gloo-gateway-single.yaml open gloo-gateway-single.yaml
-
In the Helm chart for your management cluster, add the endpoint, port, and base path for your Jaeger instance in the
glooUi
section to embed the Jaeger UI into the Gloo UI, and disable the Jaeger UI that is automatically built in to Gloo Gateway. Note that your Jaeger platform must be served from a path that is different from/
as this path is reserved for the built-in Jaeger instance in Gloo Gateway.To pick up traces from workloads and send them to your Jaeger instance, enable the built-in
traces/istio
pipeline and add an extra exporter.jaeger: enable: false glooUi: tracing: endpoint: "<endpoint>" port: <port> basePath: "/<base-path>" telemetryGateway: enabled: false telemetryCollectorCustomization: extraExporters: otlp/jaeger-custom: endpoint: myjaeger:4317 tls: insecure: true pipelines: traces/istio: enabled: true pipeline: exporters: - otlp/jaeger-custom telemetryCollector: enabled: true
-
Upgrade your installation by using your updated values file.
helm upgrade gloo-platform gloo-platform/gloo-platform \ --namespace gloo-mesh \ -f gloo-gateway-single.yaml \ --version $UPGRADE_VERSION
-
Verify that your custom Jaeger settings were added to the Gloo telemetry collector configmap.
kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml
-
Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pod.
kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent
-
Get the Helm values files for your current version.
- Get your current values for the management cluster.
helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $MGMT_CONTEXT > mgmt-server.yaml open mgmt-server.yaml
- Get your current values for the workload clusters.
helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $REMOTE_CONTEXT > agent.yaml open agent.yaml
- Get your current values for the management cluster.
-
In the Helm chart for your management cluster, add the endpoint, port, and base path for your Jaeger instance in the
glooUi
section to embed the Jaeger UI into the Gloo UI and disable the Jaeger UI that is automatically built in to Gloo Gateway. Note that your Jaeger platform must be served from a path that is different from/
as this path is reserved for the built-in Jaeger instance in Gloo Gateway.To forward traces from the Gloo telemetry gateway to your Jaeger instance, set up an extra exporter on the Gloo telemetry gateway with the details of your Jaeger endpoint, and add that exporter to the default
traces/jaeger
pipeline in your Helm values file. For more information about how to set up the exporter, see the OTLP gRPC exporter documentation.To collect traces from workloads in the management cluster, enable the
traces/istio
pipeline in the Gloo telemetry collector agent.jaeger: enable: false glooUi: tracing: endpoint: "<endpoint>" port: <port> basePath: "/<base-path>" telemetryGateway: enabled: true service: type: ClusterIP telemetryGatewayCustomization: extraExporters: otlp/jaeger-custom: endpoint: myjaeger:4317 tls: insecure: true pipelines: traces/jaeger: enabled: true pipeline: exporters: - otlp/jaeger-custom telemetryCollector: enabled: true telemetryCollectorCustomization: pipelines: traces/istio: enabled: true extraExporters: otlp/jaeger: endpoint: gloo-jaeger-collector.gloo-mesh.svc:4317 tls: insecure: true
-
In the Helm value file for the workload cluster, enable the default
traces/istio
pipeline in your Helm values file to collect traces from Istio-enabled workloads. Thetraces/istio
pipeline is configured on the Gloo collector agents and sends traces to the Gloo telemetry gateway in the management cluster.telemetryCollector: config: exporters: otlp: endpoint: gloo-telemetry-gateway.gloo-mesh:4317 telemetryCollectorCustomization: pipelines: traces/istio: enabled: true
-
Upgrade the management cluster.
helm upgrade gloo-platform gloo-platform/gloo-platform \ --kube-context $MGMT_CONTEXT \ --namespace gloo-mesh \ -f mgmt-server.yaml \ --version $UPGRADE_VERSION
-
Verify that your settings are applied in the management cluster.
-
Verify that the tracing settings were added to the Gloo telemetry collector and gateway configmaps.
kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context $MGMT_CONTEXT kubectl get configmap gloo-telemetry-gateway-config -n gloo-mesh -o yaml --context $MGMT_CONTEXT
-
Perform a rollout restart of the telemetry gateway deployment and the telemetry collector daemon set to force your configmap changes to be applied to the telemetry gateway and collector agent pods.
kubectl rollout restart -n gloo-mesh deployment/gloo-telemetry-gateway --context $MGMT_CONTEXT kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context $MGMT_CONTEXT
-
-
Upgrade the workload cluster.
helm upgrade gloo-platform gloo-platform/gloo-platform \ --kube-context $REMOTE_CONTEXT \ --namespace gloo-mesh \ -f agent.yaml \ --version $UPGRADE_VERSION
-
Verify that your settings are applied in the workload cluster.
-
Verify that the tracing settings were added to the Gloo telemetry collector configmap.
kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context $REMOTE_CONTEXT
-
Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pods.
kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context $REMOTE_CONTEXT
-
Step 3: Verify tracing
Open the Gloo UI and verify that traces are collected for your Istio workloads.
-
Open the Gloo UI.
-
From the menu, select Observability > Tracing and verify that the Jaeger UI opens.
-
Send a few sample requests to your Istio workloads. Each request produces Istio traces that are sent to the Jaeger instance that you configured. For example if you deployed the Bookinfo sample app from the Get started guide, use the following steps to produce traces.
- Port-forward the product page app.
kubectl port-forward deployment/productpage-v1 -n bookinfo 9080
- Open the product page app.
- Refresh the page multiple times.
- Port-forward the product page app.
-
Wait a few seconds and verify that traces are displayed in the Gloo UI.