About

When you use community Istio for your ambient mesh, ztunnel is configured to generate TCP metrics, logs, and traces for all service traffic in the ambient mesh. In addition, you can enable Layer 7 (L7) HTTP and HTTP/2 telemetry data when using waypoint proxies.

With the Solo distribution of Istio, L7 attributes are automatically added to ztunnel metrics, traces, and logs so that you can obtain HTTP observability in to your ambient mesh, even if no waypoint proxies are deployed. Access to L7 attributes, such as the source and destination workload IDs, response codes, HTTP methods, and paths, can help you troubleshoot, monitor, and audit traffic in your ambient mesh.

L7 observability is enabled by default in the Solo distribution of Istio. In addition, all ztunnel instances are configured to send their telemetry data to the Gloo telemetry pipeline so that you can access the data with the built-in observability tools, such as Prometheus or the Gloo UI.

View the L7 ztunnel configuration

To verify that L7 observability is enabled in your ztunnel, you can review the ztunnel configuration. Verify that access logs, metrics, and traces are enabled in your configuration. Note that this is the default setting when using the Solo distribution of Istio.

  istioctl ztunnel-config all -ojson | jq .config.l7Config

Example output:

  {
  "access_log": {
    "enabled": true,
    "skip_connection_log": false
  },
  "enabled": true,
  "metrics": {
    "enabled": true
  },
  "tracing": {
    "enabled": true,
    "otlp_endpoint": "http://opentelemetry-collector.istio-system:4317"
  }
}

Customize the configuration

If metrics, access logs, or traces are not enabled, you can enable them with the following command.

  kubectl set env ds/ztunnel -n istio-system L7_ENABLED=true

To customize the observability settings, you can use the following fields during installation of the ztunnel Helm chart.

FieldDescription
l7Telemetry.enabledGlobally enable HTTP telemetry. This field must be set to true for any telemetry data to be collected.
l7Telemetry.metrics.enabledEnable HTTP metrics collection.
l7Telemetry.accessLog.enabledEnable HTTP access logs collection.
l7Telemetry.accessLog.skipConnectionLogIf enabled, connections that have only HTTP requests record the TCP connection log at the debug level. Typically, the debug log level is disabled, in which case the TCP connection log is effectively not recorded. This can be useful for short-lived connections in which logging both TCP connections and HTTP requests causes excessive noise. Note that if the connection does not use HTTP, the TCP connection event is always logged. If this field is disabled, which is the default value, both HTTP requests and TCP connection logs are recorded.
l7Telemetry.distributedTracing.enabledEnable HTTP tracing.
l7Telemetry.distributedTracing.otlpEndpointThe OTLP endpoint to send traces to, such as http://gloo-telemetry-collector.gloo-mesh:4317 for the Gloo telemetry collector.

Example L7 telemetry data

Review examples of L7 telemetry data when using the Solo distribution of Istio.

Metrics

When enabling L7 observability for ztunnels, you get access to the following L7 Istio metrics, even if no waypoints are deployed. These metrics are enriched with L7 attributes, such as the source and destination workload IDs, response codes, and connection policies.

  • istio_requests_total: Total count of HTTP requests. The response_code label indicates the HTTP status code for the request.
  • istio_request_duration_milliseconds: Distribution of the duration of each HTTP request.
  • istio_request_bytes: Distribution of HTTP request sizes.
  • istio_response_bytes: Distribution of HTTP response sizes.

You can access these metrics by using the built-in Prometheus server. For more information, see Run sample PromQL queries.

Example L7 ztunnel metric istio_requests_total:

  {app="gloo-telemetry-collector-agent", cluster="gloo-gateway-docs-mgt", 
collector_pod="gloo-telemetry-collector-agent-mhl9r", 
component="agent-collector", 
connection_security_policy="mutual_tls", 
controller_revision_hash="7f68544cc6", 
destination_cluster="gloo-gateway-docs-mgt", 
destination_principal="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings", 
destination_service="ratings.bookinfo.svc.cluster.local", destination_workload="ratings-v1", 
destination_workload_id="ratings-v1.bookinfo.gloo-gateway-docs-mgt", 
destination_workload_namespace="bookinfo", 
namespace="istio-system", pod_template_generation="1", reporter="destination", response_code="200", 
response_flags="-", source_cluster="gloo-gateway-docs-mgt", source_principal="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-productpage", 
source_workload="productpage-v1", source_workload_namespace="bookinfo", 
workload_id="productpage-v1.bookinfo.gloo-gateway-docs-mgt"}

Access logs

Access logs are enabled by default in the Solo distribution of Istio and are automatically enriched with L7 attributes, such as the HTTP method, path, protocol, HTTP response code, host, and user agent.

In addition to logging HTTP requests, the TLS tls.sni and tls.alpn attributes are also added for TLS requests. Note that because ztunnel does not terminate TLS connections from apps, traffic is classified either as TLS or HTTP. If the traffic type is HTTPS, it is classified as TLS.

To view access logs, you simply review the logs of the ztunnel instance that is located on the same node as your source or destination workload.

  kubectl logs <ztunnel> -n istio-system

Example ztunnel log entry:

  2025-02-13T17:42:55.393871Z	info	http access	request complete	
 src.addr=10.0.1.204:51860 src.workload="productpage-v1-6cc5c84969-chgsz" src.namespace="bookinfo" 
 src.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-productpage" 
 dst.addr=10.0.7.169:15008 dst.hbone_addr=10.0.7.169:9080 dst.service="ratings.bookinfo.svc.cluster.local" 
 dst.workload="ratings-v1-856c9b5cc9-kpfh5" dst.namespace="bookinfo" 
 dst.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings" direction="outbound" method=GET 
 path="/ratings/1" protocol=HTTP1 response_code=200 host="ratings:9080" user_agent="curl/8.12.1" duration="1ms"

2025-02-13T17:42:55.395065Z	info	access	connection complete	
src.addr=10.0.1.204:51860 src.workload="productpage-v1-6cc5c84969-chgsz"
src.namespace="bookinfo" src.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-productpage"
dst.addr=10.0.7.169:15008 dst.hbone_addr=10.0.7.169:9080
dst.service="ratings.bookinfo.svc.cluster.local" dst.workload="ratings-v1-856c9b5cc9-kpfh5"
dst.namespace="bookinfo" dst.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
direction="outbound" bytes_sent=85 bytes_recv=222 duration="6ms"

Traces

When using the Solo distribution of Istio, ztunnels are automatically configured to monitor HTTP traffic and to report traces to the tracing backend.

Note that the ztunnel cannot initiate new tracing spans. You must route your request through an ingress gateway or a waypoint proxy to initiate the tracing span. Because these requests are also routed through the ztunnel, the ztunnel can report back traces to the Gloo telemetry collector agent. You can view the traces in the built-in Jaeger frontend in the Gloo UI.

To enable traces for your Istio components, see Add Istio request traces.