About

When you use community Istio for your ambient mesh, ztunnel is configured to generate TCP metrics, logs, and traces for all service traffic in the ambient mesh. In addition, you can enable Layer 7 HTTP and HTTP/2 telemetry data when using waypoint proxies.

With the Solo distribution of Istio, Layer 7 attributes are automatically added to ztunnel metrics, traces, and logs so that you can obtain HTTP observability in to your ambient mesh, even if no waypoint proxies are deployed. Access to Layer 7 attributes, such as the source and destination workload IDs, response codes, HTTP methods, and paths, can help you troubleshoot, monitor, and audit traffic in your ambient mesh.

Layer 7 observability is enabled by default in the Solo distribution of Istio. In addition, all ztunnel instances are configured to send their telemetry data to the Gloo telemetry pipeline so that you can access the data with the built-in observability tools, such as Prometheus or the Gloo UI.

View the L7 ztunnel configuration

To verify that Layer 7 observability is enabled in your ztunnel, you can review the ztunnel configuration. Verify that access logs, metrics, and traces are enabled in your configuration. Note that this is the default setting when using the Solo distribution of Istio.

  istioctl ztunnel-config all -ojson | jq .config.l7Config

Example output:

  {
  "access_log": {
    "enabled": true,
    "skip_connection_log": false
  },
  "enabled": true,
  "metrics": {
    "enabled": true
  },
  "tracing": {
    "enabled": true,
    "otlp_endpoint": "http://opentelemetry-collector.istio-system:4317"
  }
}

If metrics, access logs, or traces are not enabled, you can enable them with the following command.

  kubectl set env ds/ztunnel -n istio-system L7_ENABLED=true

Example L7 telemetry data

Review examples of Layer 7 telemetry data when using the Solo distribution of Istio.

Metrics

When enabling Layer 7 observability for ztunnels, you get access to Layer 7 Istio metrics, such as istio_requests_total, istio_request_duration_milliseconds, istio_request_bytes, and istio_response_bytes, even if no waypoints are deployed. These metrics are enriched with Layer 7 attributes, such as the source and destination workload IDs, response codes, and connection policies.

You can access these metrics by using the built-in Prometheus server. For more information, see Run sample PromQL queries.

Example Layer 7 ztunnel metric istio_requests_total:

  {app="gloo-telemetry-collector-agent", cluster="gloo-gateway-docs-mgt", 
collector_pod="gloo-telemetry-collector-agent-mhl9r", 
component="agent-collector", 
connection_security_policy="mutual_tls", 
controller_revision_hash="7f68544cc6", 
destination_cluster="gloo-gateway-docs-mgt", 
destination_principal="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings", 
destination_service="ratings.bookinfo.svc.cluster.local", destination_workload="ratings-v1", 
destination_workload_id="ratings-v1.bookinfo.gloo-gateway-docs-mgt", 
destination_workload_namespace="bookinfo", 
namespace="istio-system", pod_template_generation="1", reporter="destination", response_code="200", 
response_flags="-", source_cluster="gloo-gateway-docs-mgt", source_principal="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-productpage", 
source_workload="productpage-v1", source_workload_namespace="bookinfo", 
workload_id="productpage-v1.bookinfo.gloo-gateway-docs-mgt"}

Access logs

Access logs are enabled by default in the Solo distribution of Istio and are automatically enriched with Layer 7 attributes, such as the HTTP method, path, protocol, HTTP response code, host, and user agent.

To view access logs, you simply review the logs of the ztunnel instance that is located on the same node as your source or destination workload.

  kubectl logs <ztunnel> -n istio-system

Example ztunnel log entry:

  2025-02-13T17:42:55.393871Z	info	http access	request complete	
 src.addr=10.0.1.204:51860 src.workload="productpage-v1-6cc5c84969-chgsz" src.namespace="bookinfo" 
 src.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-productpage" 
 dst.addr=10.0.7.169:15008 dst.hbone_addr=10.0.7.169:9080 dst.service="ratings.bookinfo.svc.cluster.local" 
 dst.workload="ratings-v1-856c9b5cc9-kpfh5" dst.namespace="bookinfo" 
 dst.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings" direction="outbound" method=GET 
 path="/ratings/1" protocol=HTTP1 response_code=200 host="ratings:9080" user_agent="curl/8.12.1" duration="1ms"

2025-02-13T17:42:55.395065Z	info	access	connection complete	
src.addr=10.0.1.204:51860 src.workload="productpage-v1-6cc5c84969-chgsz"
src.namespace="bookinfo" src.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-productpage"
dst.addr=10.0.7.169:15008 dst.hbone_addr=10.0.7.169:9080
dst.service="ratings.bookinfo.svc.cluster.local" dst.workload="ratings-v1-856c9b5cc9-kpfh5"
dst.namespace="bookinfo" dst.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
direction="outbound" bytes_sent=85 bytes_recv=222 duration="6ms"

Traces

When using the Solo distribution of Istio, ztunnels are automatically configured to monitor HTTP traffic and to report traces to the tracing backend.

Note that the ztunnel cannot initiate new tracing spans. You must route your request through an ingress gateway or a waypoint proxy to initiate the tracing span. Because these requests are also routed through the ztunnel, the ztunnel can report back traces to the Gloo telemetry collector agent. You can view the traces in the built-in Jaeger frontend in the Gloo UI.

To enable traces for your Istio components, see Add Istio request traces.