For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Review recommendations for taking ambient mesh to production, covering resource allocation, scaling, and high-availability configuration.

Configuration changes for production

Resource allocation and scaling

Ztunnel is a DaemonSet, which runs one pod per node. Ztunnel is designed to have a small footprint and to handle large-scale clusters out of the box, scaling with cluster size (pods and services) and traffic rates (connections, requests, and throughput).

Ztunnel configuration changes are typically not required unless one of the following conditions applies:

• The cluster has over 100,000 pods or 20,000 services.
• An individual node serves over 20,000 connections, 100,000 requests per second, or 5Gb/s of traffic.

These are not limits of ztunnel, which can scale beyond these thresholds. They are the recommended point at which you might analyze usage and consider vertically scaling the CPU and memory reservations to match observed usage.

Ztunnel runs with a small CPU and memory reservation by default, and no limits. However, its worker thread count caps its maximum CPU usage.

In Istio 1.31 and later, ztunnel sizes its worker thread pool from the CPU resources of its own pod. CPU quantities round up to whole cores, so 500m gives 1 core, 1.5 gives 2, etc.

• If you set neither a CPU request nor limit, ztunnel uses 25% of the logical cores of the node, with a minimum of 2 threads.
• If you set only a CPU request, ztunnel uses 25% of the logical cores of the node or the request size, whichever is greater, with a minimum of 2 threads. The chart requests 200m by default, which rounds to 1 core and is therefore too small to raise the count.
• If you set a CPU limit, ztunnel uses that limit as its thread count, or the CPU request if that is higher (not recommended).

In earlier versions, ztunnel always ran 2 worker threads unless you overrode the count by using the ZTUNNEL_WORKER_THREADS environment variable. On nodes with more than 8 cores, expect the ztunnel worker thread count to rise after you upgrade to 1.31.

To inspect the current thread count:

istioctl ztunnel-config all -ojson | jq .config.numWorkerThreads

Review the following resource limit recommendations:

• CPU limits: Do not set CPU limits on the pod. Ztunnel does not need a limit to bound its CPU usage, because the worker thread count already does this. If you must set a limit, do not set it to 1 core or less, which takes ztunnel below 2 worker threads.
• Memory limits: Do not set memory limits. If you must set one, set it as high as possible. If the limit is exceeded simultaneously across nodes, all ztunnel pods could be killed and cause a cluster-wide networking outage.

Tune the ztunnel worker thread count

The automatic sizing based on node resources described above is a good default. Override it when you want a thread count that does not follow from the CPU resources of the pod, such as a fixed count across a fleet of differently sized nodes.

The ZTUNNEL_WORKER_THREADS environment variable on the ztunnel pod takes precedence over the automatic sizing. To tune the count, use one of the following methods:

• Set a CPU request: Set resources.requests.cpu in the ztunnel chart. The request reserves the CPU and sets the minimum thread count in one value. This is the best option for most clusters.
• Set an explicit thread count: Set ZTUNNEL_WORKER_THREADS to a fixed number such as 8, to a percentage such as 25% (one thread per four cores, useful in mixed-node environments), or to 0 to use all available cores. Percentages and 0 are relative to the CPU limit of the pod when one is set, and to the logical cores of the node otherwise. A fixed number is not limit-aware: setting more threads than the CPU limit allows can result in substantial performance degradation.

Upgrading

To understand how to safely upgrade Istio when a new version is released, see the upgrading guide.

Component health monitoring

Monitor ambient mesh components at the default log level. Only increase log verbosity when actively troubleshooting, because debug logging under production traffic generates significant volume.

ztunnel

Ztunnel runs as a DaemonSet, so an unhealthy pod on any node means workloads on that node lose mesh connectivity. Check pod status across all nodes.

kubectl get pods -n istio-system -l app=ztunnel

Metrics

Ztunnel exposes Prometheus metrics on port 15020. The Istio sample Grafana dashboards include a ztunnel dashboard covering pod CPU and memory usage, connection counts, and bytes transmitted. Monitor for unexpected spikes or drops in any of these values.

Key metrics to alert on include:

• istio_tcp_connections_opened_total: Rate of new connections through ztunnel. A sudden drop on a node indicates ztunnel may have stopped processing traffic.
• istio_tcp_sent_bytes_total / istio_tcp_received_bytes_total: Bytes flowing through ztunnel. A sustained drop relative to your baseline suggests traffic is no longer routing through the mesh.

Logs

Ztunnel writes an access log entry for each connection on completion (not on open). Each entry includes source and destination workload identity, bytes transferred, duration, and an error field when the connection did not complete normally.

To surface errors across all ztunnel pods:

kubectl logs -n istio-system -l app=ztunnel --prefix | grep error

Common error patterns to watch for:

• TLS handshake error: Certificate or identity issue. Check that both workloads are enrolled in the mesh and that certificates have not expired.
• io error: deadline has elapsed: The connection timed out. Typically a network policy or firewall rule is blocking traffic between pods or to the destination.
• no healthy upstream: The destination has no ready endpoints. Check the target pod’s health and whether it is enrolled in the mesh.
• io error: Cannot assign requested address: A local network configuration problem on the node, often seen during node instability.

For common error scenarios and remediation steps, see the Istio ambient troubleshooting wiki.

istiod

An unhealthy istiod prevents configuration from reaching ztunnel and waypoint proxies, which causes policy and routing to become stale. Check the status of the istiod pod.

kubectl get pods -n istio-system -l app=istiod

Metrics

The Istio sample Grafana dashboards include a Pilot dashboard covering xDS push health, push latency, and connected proxy counts. Key metrics to alert on include:

• pilot_xds_rejects: Configurations rejected by connected proxies. Any non-zero value indicates a configuration error that proxies refused to accept.
• pilot_xds_connected_proxies: Number of ztunnel and waypoint proxies connected to the control plane. A sudden drop means proxies have lost contact with istiod.
• pilot_proxy_convergence_time: Time for configuration changes to propagate to all proxies. A sustained increase indicates control plane stress from high churn or large scale.

Logs

Under normal operation, istiod logs are primarily info-level xDS push activity. Watch for warn and error messages:

kubectl logs -n istio-system -l app=istiod | grep -iE 'warn|error'

Repeated warnings or errors referencing rejected configurations, certificate signing failures, or proxy disconnects are worth investigating promptly.

Applications

Adding workloads to the mesh does not change how you monitor your applications. Continue watching the same signals as before: request latency, error rates (4xx/5xx), pod readiness, and replica counts. If you observe unexpected changes in these signals after enrolling a namespace in the mesh, check the ztunnel logs on the affected nodes first.

Node readiness

In production environments with autoscaling or frequent node replacements, Kubernetes can schedule pods onto a new node before the istio-cni agent is ready, leaving those pods without traffic redirection. To prevent this race condition, configure the Istio untaint controller to hold pods with a node startup taint until CNI initialization completes. For details, see Using the untaint controller.

Advanced ztunnel performance tuning

Beyond the resource allocation guidance above, Solo Enterprise for Istio provides additional controls for tuning ztunnel performance in large-scale deployments, including worker thread configuration, connection queue tuning, and per-node traffic shaping. For details, see Tune ztunnel performance.

Certificate management

In production environments, using the default self-signed Istio CA is not recommended. Solo Enterprise for Istio supports plugging in an external certificate authority so that workload certificates are issued from your organization’s PKI.

• For an overview of Istio’s certificate hierarchy and the challenges of multicluster trust, see Istio certificate management.
• To issue Istio intermediate CA certificates using AWS Private Certificate Authority and cert-manager, see Bring your own Istio CAs with AWS.

Certificate revocation list (CRL) enforcement

Solo Enterprise for Istio supports enforcing a Certificate Revocation List on ztunnel outbound connections, so that peers presenting revoked certificates are rejected at the data plane. For details, see CRL enforcement.

Cluster draining

To gracefully decommission a cluster or prepare it for maintenance, you can mark it as draining so that no new connections are routed to it. Solo Enterprise for Istio supports this through the solo.io/draining-weight annotation on east-west gateways. For details, see Drain clusters in the mesh.

Enterprise observability with Solo UI

The health monitoring guidance above references community Grafana dashboards. With an Enterprise license, you can also deploy the Solo UI, which provides a visual service graph with live metrics, a resource overview for all Istio and Kubernetes resources, and global service visibility across clusters, all in a single interface. For telemetry storage and querying, the Solo UI integrates with ClickHouse.