About the telemetry pipeline
Learn about the telemetry pipeline architecture, its components, and default pipelines that you can choose from.
You can gain insights into the health and performance of your cluster components by using the telemetry pipeline. Built on top of the OpenTelemetry open source project, the telemetry pipeline helps you to collect and export telemetry data, such as metrics, logs, traces, and Solo insights, and to visualize this data by using built-in observability tools.
Review the information on this page to learn more about the telemetry pipeline and how to use it in your environment.
Setup
The telemetry pipeline is set up by default when you install the Solo UI. To see the receivers, processors, and exporters that are set up by default for you, run the following command.
kubectl get configmap solo-enterprise-telemetry-collector-config -n solo-enterprise -o yamlArchitecture
The telemetry pipeline is built around the solo-enterprise-telemetry-collector pod, which runs an OpenTelemetry collector. All telemetry data, including metrics, traces, logs, and Kubernetes object snapshots, is stored in the ClickHouse instance deployed alongside the Solo UI. The Solo UI reads directly from ClickHouse to power the service graph, metrics views, and resource overviews. For a full component diagram and port reference, see Architecture.
Single cluster setup
In a single cluster setup, the telemetry collector scrapes Istio component metrics from the cluster using a built-in Prometheus receiver, targeting istiod, ztunnel, and waypoint proxy pods. It also receives OTLP telemetry data, such as traces and metrics, from local mesh workloads. The collector enriches data with Kubernetes metadata, filters it to reduce cardinality, and writes all telemetry to ClickHouse.
Multicluster setup
In a multicluster setup, a telemetry collector is deployed in each cluster, and the solo-enterprise-ui pod in the management cluster also runs a telemetry gateway.
In the management cluster, the telemetry collector performs the same local scraping and enrichment as in a single cluster setup. The telemetry gateway receives telemetry from workload cluster collectors on port 4316 via the otlp/remote receiver, then writes that data to ClickHouse.
In workload clusters, the relay Helm chart deploys a telemetry collector that sends all collected data to the telemetry gateway in the management cluster over port 4316.
Telemetry pipelines
The telemetry collector runs the following built-in pipelines. All pipelines write to ClickHouse.
| Pipeline | Receivers | Setup | Description |
|---|---|---|---|
traces/istio | otlp/local, otlp/remote | Single and multicluster | Collects Istio request traces from local mesh workloads and, in multicluster setups, from workload cluster collectors. The filter/istio processor retains only spans from Envoy and ztunnel instrumentation scopes. |
metrics/istio | prometheus | Single and multicluster | Scrapes Istio component metrics directly from istiod, ztunnel, and waypoint proxy pods using a Prometheus receiver. The filter/istio_metrics processor limits collection to a curated set of Istio metrics, and the k8sattributes and transform/health_metrics processors enrich metrics with Kubernetes metadata and reporter labels. For the list of metrics collected, see Default metrics in the pipeline. |
metrics/otlp | otlp/local, otlp/remote | Single and multicluster | Collects OTLP metrics pushed from local mesh workloads and, in multicluster setups, from workload cluster collectors. |
logs/events | otlp/local | Single and multicluster | Collects Kubernetes events from the local cluster. |
logs/in | otlp/remote | Multicluster only | Receives logs from workload cluster collectors and routes them to either logs/remoteobjects or logs/remoteevents based on the k8sobjects attribute. |
logs/remoteobjects | routing/k8sobjects connector | Multicluster only | Receives Kubernetes object snapshots routed from logs/in and writes them to ClickHouse. |
logs/remoteevents | routing/k8sobjects connector | Multicluster only | Receives Kubernetes events routed from logs/in and writes them to ClickHouse. |
Metrics
For details on the Istio metrics that the telemetry pipeline collects by default, how they are enriched, and the full list of metric labels, see Metrics.
Traces
The traces/istio pipeline collects Istio request traces from mesh workloads and stores them in ClickHouse. Traces are not currently displayed in the Solo UI, but you can configure Istio to send traces directly to your own Jaeger instance for visualization. For more information, see Collect Istio request traces.
ClickHouse
All telemetry data is written to a ClickHouse instance deployed alongside the Solo UI. Data is retained for 90 days. You can query ClickHouse directly or connect Grafana to build custom dashboards. For more information, see ClickHouse data store.