Tune ztunnel performance

Tune ztunnel performance settings for large-scale ambient mesh deployments.

Use the settings in this guide to tune ztunnel’s behavior at scale. For example, you can control how long ztunnel drains before restarting by adjusting the termination grace period, and reduce memory pressure from stale connections with TCP keepalive probes.

Configure the termination grace period

When Kubernetes restarts a ztunnel pod during an upgrade, node eviction, or any other pod lifecycle event, ztunnel starts draining active connections. The terminationGracePeriodSeconds value controls how long ztunnel has to complete that drain before Kubernetes sends a SIGKILL. Any TCP connection that outlives the grace period is dropped at the node.

The default Kubernetes value is 30 seconds. For clusters where workloads maintain long-lived connections, such as streaming RPCs or persistent database connections, increasing the grace period reduces the chance that a ztunnel restart drops a connection that is still active.

Apply the grace period to your ztunnel installation.

helm upgrade ztunnel oci://us-docker.pkg.dev/soloio-img/istio-helm/ztunnel \
  --version 1.30.3-solo \
  --namespace istio-system \
  --reuse-values \
  --set terminationGracePeriodSeconds=300

Verify that the DaemonSet updated successfully.

kubectl -n istio-system rollout status daemonset/ztunnel

Warning

A longer grace period increases the time that a node upgrade or eviction takes to complete. For clusters using blue/green node pools, the grace period also affects how quickly drained nodes can be terminated. Balance the connection longevity requirements in your environment against the operational overhead of longer node drain times.

Configure TCP keepalive probes

About FIN_WAIT2 connection accumulation

In ambient mesh deployments at scale, ztunnel can accumulate connections in the Linux FIN_WAIT2 state. A connection enters FIN_WAIT2 when the local side has sent a FIN to close the connection but the remote side has not yet acknowledged the close. When workloads terminate TCP connections abruptly, such as by killing a process without a graceful shutdown, these half-closed connections persist in the kernel’s TCP state table. At high connection rates, this accumulation steadily increases ztunnel’s memory footprint and can trigger an OOMKill on the affected node over time.

Ztunnel enables TCP keepalive probes by default. When a probe goes unanswered, the kernel tears down the socket. At the default 180-second probe interval, stale connections are cleaned up within several minutes. In environments with a high rate of short-lived or misbehaving connections, lowering the interval speeds up cleanup. Enabling userTimeoutEnabled alongside keepalives closes connections that stop responding mid-stream, not just idle ones.

Diagnose FIN_WAIT2 accumulation

Before tuning keepalive settings, confirm that FIN_WAIT2 connections are accumulating on your nodes.

Open a shell on a node where ztunnel is running.
Count the connections in FIN_WAIT2 state.
```
ss -tnH state fin-wait-2 | wc -l
```

A count that grows continuously without decreasing, combined with rising ztunnel memory usage across the same period, indicates that stale connections are accumulating.

Configure TCP keepalive tuning

The Solo ztunnel Helm chart (oci://us-docker.pkg.dev/soloio-img/istio-helm/ztunnel) exposes keepalive settings under socketConfig. The following values lower the probe interval from the 180-second default to 90 seconds and enable userTimeoutEnabled. Adjust the values based on your environment.

Field	Default	Description
`socketConfig.keepaliveEnabled`	`true`	Enables TCP keepalive probes on all ztunnel-managed connections. Keepalives are on by default.
`socketConfig.userTimeoutEnabled`	`false`	Enables the `TCP_USER_TIMEOUT` socket option, which closes connections that stop receiving ACKs within the keepalive timeout window. Recommended alongside keepalive tuning.
`socketConfig.keepaliveTime`	`secs: 180, nanos: 0`	How long a connection must be idle before the kernel sends the first keepalive probe.
`socketConfig.keepaliveInterval`	`secs: 180, nanos: 0`	How long the kernel waits between successive keepalive probes when no response is received.
`socketConfig.keepaliveRetries`	`9`	Number of unanswered probes before the kernel closes the connection.

Create a Helm values file with the keepalive tuning.

cat <<EOF > ztunnel-keepalive.yaml
socketConfig:
  keepaliveEnabled: true
  userTimeoutEnabled: true
  keepaliveTime:
    secs: 90
    nanos: 0
  keepaliveInterval:
    secs: 90
    nanos: 0
  keepaliveRetries: 4
EOF

Upgrade your ztunnel installation to apply the values.

helm upgrade ztunnel oci://us-docker.pkg.dev/soloio-img/istio-helm/ztunnel \
  --version 1.30.3-solo \
  --namespace istio-system \
  -f ztunnel-keepalive.yaml \
  --reuse-values

Verify that ztunnel picked up the updated configuration. After the rollout completes, ztunnel logs the active socket configuration at startup.
```
kubectl logs -n istio-system -l app=ztunnel --tail=100 | grep -i keepalive
```

After applying the changes, the FIN_WAIT2 count on affected nodes should stabilize and gradually decrease as the kernel times out stale connections at the new interval.

Drain clusters in the mesh Uninstall