For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Review summaries of the main changes in the Solo Enterprise for Istio 1.31 release.

The release notes include important installation changes and known issues. They also highlight ways that you can take advantage of new features or enhancements to improve your product usage.

For more information, see the following related resources:

• Ambient or sidecar mesh upgrade guides: Steps to upgrade your service meshes to the latest Istio version supported by Solo Enterprise for Istio.
• Solo UI upgrade guide: Steps to upgrade the Solo UI to the latest supported version.
• Version reference: Information about Solo’s version support.

🔥 Breaking changes

Review details about the following breaking changes. The severity is intended as a guide to help you assess how much attention to pay to this area during the upgrade, but can vary depending on your environment.

🚨 High

Review severe changes that can impact production and require manual intervention.

• No high-severity changes are currently reported.

🔔 Medium

Review changes that might have impact to production and require manual intervention, but possibly not until the next version is released.

ztunnel L7 access logs and distributed tracing disabled by default

In 1.31, two ztunnel Helm chart values default to false instead of true:

L7 metrics (istio_requests_total and related metrics) and TCP connection logs are unchanged.

If you consume ztunnel HTTP request logs or ztunnel trace spans, you must explicitly include these settings and set them to enabled in your ztunnel Helm values file before you upgrade.

l7Telemetry:
  accessLog:
    enabled: true
  distributedTracing:
    enabled: true
    otlpEndpoint: "http://<your-collector>:4317"

If you omit these values before upgrading, ztunnel HTTP request log lines stop appearing and ztunnel spans stop reaching your tracing backend. Traffic is not affected. Applying these values and upgrading again restores telemetry. No restart is required because ztunnel reloads its L7 configuration in place.

ℹ️ Low

Review informational updates that you might want to implement but that are unlikely to materially impact production.

ingress_use_waypoint field removed from SoloExtensions Envoy filter

The ingress_use_waypoint field on the SoloExtensions Envoy filter configuration is removed in 1.31. Use the upstream istio.io/ingress-use-waypoint: "true" label on the Service resource instead. If you set ingress_use_waypoint directly on a SoloExtensions EnvoyFilter, remove that configuration and apply the label to the relevant Service resources before upgrading.

🚧 New known issues

Canary upgrades with the Gateway API cause brief traffic blackhole on tag flip

When Gateway API resources manage ingress traffic, flipping the default revision tag to a new istiod causes the old control plane to push empty xDS to Gateway pods that are still connected to it. Because the new istiod has not yet connected, Gateway pods have no routing rules for approximately 15–30 seconds, blackholing all Gateway-managed ingress traffic.

The upstream fix is available in Istio 1.29.5 and later, 1.30.2 and later, and all 1.31.x patch versions. For workarounds, see Canary upgrades with the Gateway API.

WorkloadEntry autoregistration omits HBONE tunnel label

🌟 New features

Review the following new features that are introduced in version 1.31 and that you can enable in your environment.

Istio 1.31 support

You can now run Solo Enterprise for Istio with Istio 1.31. For more information, see the version support matrix, and the Solo distribution of Istio changelog for 1.31.

Ambient mesh load balancing and failover documentation

New documentation is available for configuring load balancing and failover in ambient mesh deployments. The documentation covers L4 load balancing and failover with ztunnel, L7 load balancing and failover with waypoint proxies and DestinationRules, and load balancing for external traffic entering the mesh through kgateway. You can also review multicluster zone and region-aware failover for global services, including component-specific reference pages that describe how load balancing, outlier detection, and failover behave for each mesh component.

To get started, see the load balancing and failover overview.

Coexistence and migration between Istio and agentgateway waypoints (alpha)

Istio waypoints and agentgateway waypoints can now run in the same namespace and mesh without interference. Each service routes exclusively through its own configured waypoint, and policies attached to one waypoint do not apply to traffic through the other.

If you want to migrate from Istio to agentgateway waypoints, two paths are available: a label flip that switches a service to the agentgateway waypoint in one operation, and a weighted canary that gradually shifts a configurable percentage of in-mesh connections to the agentgateway waypoint while keeping the rest on the Istio waypoint. The weighted canary reuses the new istio.io/use-waypoint-canary and istio.io/use-waypoint-canary-weight fields from the upstream canary waypoint feature.

To get started, see Migrate from Istio waypoints to agentgateway waypoints.

CRL enforcement for existing ztunnel connections

Previously, CRL enforcement applied only to new inbound and outbound connections. Ztunnel now also immediately terminates any existing connections whose certificates appear on the revocation list when the CRL is reloaded. This termination is abrupt by design, because a revoked certificate is treated as a security event.

To get started, see CRL enforcement.

ECS cluster-level domain override

You can now apply the ecs.solo.io/domain tag directly to an ECS cluster resource to set a default domain for all services and tasks in that cluster. Previously, the domain that was used to construct ECS service hostnames was set only at the account level through ECS_ACCOUNTS or overridden per service with the ecs.solo.io/hostname tag. The new cluster tag fills the gap between account-level and service-level control. The domain resolution precedence is: ecs.solo.io/hostname service tag > ecs.solo.io/domain cluster tag > account-level domain > default mesh peering domain.

To get started, see Extend the mesh to ECS.

Global HTTP retry policy for ambient mesh

meshConfig.defaultHttpRetryPolicy now applies to inbound routes on waypoint proxies. Previously, this setting only affected sidecar outbound routes, which meant that ambient mesh workloads could not use a mesh-wide default retry policy. Waypoints now apply the global retry policy to inbound HTTP routes, consistent with how sidecar proxies apply it on outbound routes.

To get started, see HTTP retries.

Limit ServiceEntry scope per namespace

The new meshConfig.serviceEntryVisibility field lets mesh administrators control the visibility of a ServiceEntry across namespaces through the following settings:

• defaultVisibility: By default, a ServiceEntry has cluster-wide effect. Setting defaultVisibility: NAMESPACE restricts each ServiceEntry to workloads in its own namespace, so that workloads in other namespaces cannot resolve or reach it. Additionally, istiod does not bind the ServiceEntry to a waypoint in a different namespace. For solo-ztunnel-egress waypoint gateways, this restriction is relaxed: because ztunnel resolves each connection by using the client’s HBONE source identity, the cross-namespace binding to a solo-ztunnel-egress gateway does not expose the ServiceEntry outside its own namespace, provided that the gateway’s allowedRoutes allow the ServiceEntry’s namespace.
• policies: The policies list applies different visibility settings to namespaces that match a label selector, using first-match-wins evaluation.

To get started, see Limit ServiceEntry scope.

Mesh-wide default circuit breaker policy

A new meshConfig.defaultTrafficPolicy field lets mesh operators set circuit breaker and connection pool defaults for traffic that originates from any workload in the mesh, whether destined for another in-mesh service or an external endpoint. The field accepts a v1alpha3.TrafficPolicy value and supports the same connectionPool, outlierDetection, and other traffic policy settings available in DestinationRule. The defaults also apply to outbound traffic to destinations not registered in the service registry. Per-host DestinationRule resources inherit the mesh-wide defaults and can override individual fields without replacing the entire policy.

To get started, see Circuit breakers.

Namespace-independent hostnames for global services

Segments now support a spec.hostnames field that adds additional DNS names as first-class hosts on the generated ServiceEntry, independent of the originating namespace. Unlike spec.aliases, which create annotation-only DNS aliases that keep the original Service VIP, spec.hostnames entries appear directly in ServiceEntry.spec.hosts and are reachable as distinct hosts. Patterns follow RFC 6570 Level 1 URI Template syntax with the same {service}, {namespace}, {segmentDomain}, {labels['key']}, and {annotations['key']} variables supported by spec.aliases.

To get started, see Set up namespace-independent hostnames.

SPIFFE trust domain chain verification

A new opt-in security mode, VALIDATE_SPIFFE_TRUST_DOMAIN_NAMES=STRICT, instructs ztunnel to walk the verified mTLS certificate chain and require at least one ancestor certificate to carry a SAN that matches the leaf SPIFFE URI’s trust domain. The check applies to both inbound and outbound connections and prevents trust domain spoofing when multiple intermediate authorities share a common root of trust. The default value is OFF and existing behavior is unchanged.

To get started, see SPIFFE trust domain chain verification.

Weighted waypoint canaries

Ambient mesh now supports weighted waypoint canaries. A service or namespace can reference a primary waypoint and a canary waypoint at the same time. The istio.io/use-waypoint-canary label names the canary waypoint, the optional istio.io/use-waypoint-canary-namespace label specifies its namespace, and the istio.io/use-waypoint-canary-weight annotation sets the traffic percentage to direct to the canary, as an integer from 0 to 100. The primary waypoint handles the remaining traffic. Clients need no changes.

Apply canary configuration at the service level or at the namespace level. Services in the namespace inherit the configuration using the same rules as istio.io/use-waypoint. If istiod cannot resolve the canary waypoint, it reports an error on the binding status and routes all traffic to the primary waypoint.

Note that in flat-network multicluster setups, the peering controller exports the canary waypoint alongside the primary waypoint to the generated ServiceEntry in remote clusters and auto-peers it. When a global service (solo.io/service-scope: global) exists in more than one cluster, set the canary labels and annotation on the Service in every cluster.

To get started, see Canary waypoints.

Workload Identity Token available via metadata server

The ztunnel metadata server’s /self-identity endpoint now includes the workload’s Workload Identity Token (WIT) in its response when WIMSE workload claims are enabled (ENABLE_WORKLOAD_CLAIMS=true). Applications can use the WIT to authenticate themselves to external services, or carry workload identity across trust boundaries by passing the token as the Workload-Identity-Token HTTP header. When workload claims are not enabled or istiod has not yet minted a WIT, the endpoint returns the SPIFFE identity alone with no error.

To get started, see Query workload identity from applications.

x5c trust domain verification for Workload Identity Tokens

When VALIDATE_SPIFFE_TRUST_DOMAIN_NAMES=STRICT is set on Solo Enterprise for agentgateway, and a request carries a Workload Identity Token (WIT), agentgateway now also verifies the x5c certificate chain embedded in the WIT. Agentgateway walks the x5c chain from the signing certificate up to the configured root, and rejects the request if no ancestor certificate carries a SAN that matches the SPIFFE trust domain declared in the WIT. Requests without a WIT are not affected by this check.

To get started, see Workload identity tokens (WIMSE).

ztunnel performance tuning guide

A new operations guide covers two ztunnel performance settings for large-scale ambient mesh deployments. The TCP keepalive section explains how to diagnose accumulated FIN_WAIT2 connections that cause memory pressure at scale and how to lower the probe interval via the socketConfig field. The termination grace period section describes how to reduce connection drops during ztunnel pod restarts by increasing terminationGracePeriodSeconds.

To get started, see Tune ztunnel performance.

🔄 Feature changes

Review the following changes that might impact how you use certain features in your environment.

Ambient Address xDS pushes now scoped to affected waypoints

Previously, ambient Address xDS pushes were broadcast to all connected waypoints and proxies, regardless of whether a given waypoint handled the updated workload. Pushes are now targeted to only the waypoints that are affected by the change. This reduces unnecessary xDS churn in large-scale ambient deployments. The feature is controlled by the AMBIENT_SCOPED_ADDRESS_PUSHES environment variable, which defaults to true.

istioctl multicluster link --reconcile removes orphaned Gateways

Previously, istioctl multicluster link only added Gateways and never removed them. When clusters are re-provisioned under new names, stale istio-remote-peer-* Gateways pointing at old east-west gateway IP addresses accumulated silently, reporting Programmed: True while pointing to unreachable endpoints. The new --reconcile flag removes istio-remote Gateways from remote clusters that are no longer referenced by any peering configuration. Without --reconcile, the command retains its existing additive-only behavior.

ztunnel /metrics endpoint now supports gzip compression

Ztunnel now honors Accept-Encoding: gzip requests on the /metrics Prometheus endpoint and returns a gzip-compressed response with the Content-Encoding: gzip header. Clients that do not send Accept-Encoding: gzip continue to receive uncompressed responses. This reduces cross-zone Prometheus scrape traffic in large multi-zone deployments.

🐛 Bug fixes

Agentgateway waypoints now attach HTTPRoutes to wildcard-host DYNAMIC_DNS ServiceEntries

An HTTPRoute attached to a ServiceEntry whose hosts field contains a wildcard pattern (for example, *.example.com) was silently dropped on agentgateway waypoints. The enterprise waypoint parent resolver filtered out all wildcard hosts before creating the route service key. The filter is now gated on resolution: DYNAMIC_DNS, the only resolution type where istiod allocates a VIP for a wildcard host. HTTPRoutes attached to DYNAMIC_DNS wildcard-host ServiceEntry resources now attach correctly.

EC2 ambient auto-created workload Service is now headless

When EC2 ambient discovery auto-creates a Kubernetes Service to front a discovered EC2 WorkloadEntry, the Service was incorrectly assigned a ClusterIP. In-cluster ambient pods could not reach EC2-discovered workloads through their Service, while EC2-to-EC2 traffic was unaffected. The Service is now correctly auto-created as headless (clusterIP: None), restoring connectivity from in-cluster pods to EC2 workloads.

EC2 ambient discovery no longer enters a tight reconcile loop at steady state

The EC2 discovery controller repeatedly deleted and re-created WorkloadEntry and Gateway resources for stable, unchanged EC2 workloads, generating network gateway ref not found and services already exists log messages at approximately 100 entries per second. The reconcile loop now correctly identifies unchanged workloads and does not reprocess them.

Sidecar clients no longer get stale config after ServiceEntry becomes WaypointBound

In 1.30, when a sidecar proxy was already connected to istiod at the time a ServiceEntry’s WaypointBound status was established, istiod did not push the updated route to the connected sidecar. The sidecar retained an inlined route instead of the correct tunnel-to-waypoint route, causing 404 NR route_not_found errors until the proxy reconnected. Istiod now correctly marks connected sidecar proxies dirty when a ServiceEntry’s waypoint binding is established, and pushes the updated configuration without requiring a proxy restart.

Sidecar routing updated when a service gains waypoint attachment

When ENABLE_WAYPOINT_INTEROP is enabled, a sidecar client now receives updated routing configuration when its Kubernetes service gains waypoint attachment at runtime. Previously, the waypoint-binding transition was delivered only as Endpoints and Address updates. The CDS and RDS push checks skipped these updates, leaving the sidecar routing directly to the service instead of tunneling through the waypoint and requiring a proxy restart to recover. Istiod now emits a ServiceEntry config update on every waypoint bind, unbind, or rebind transition, triggering CDS and RDS regeneration for affected sidecars.

solo.io/service-inheritance annotation now correctly removed via patch

In 1.30, patching an existing policy resource to change the solo.io/service-inheritance annotation from "true" to "false" was silently ignored. The inherited policy remained attached to the global ServiceEntry indefinitely. The annotation update path is now re-evaluated on reconcile, so patching the annotation to "false" correctly detaches the policy from the global service.

solo.io/sidecar-skip-waypoint annotation now honored for DNS ServiceEntry resources

In 1.30, the solo.io/sidecar-skip-waypoint: "true" annotation was applied only in the CDS cluster-key path and ignored in the EDS endpoint builder. For a ServiceEntry with resolution: DNS bound to a waypoint, sidecar proxies were still routed through the waypoint regardless of the annotation. The EDS endpoint builder now checks the annotation, so sidecar proxies correctly bypass the waypoint for DNS-type service entries.

ListenerSet and XListenerSet duplicate-hostname conflicts now reflected in status

When two ListenerSet or XListenerSet resources declared the same hostname on the same port, istiod silently dropped one listener. However, both resources continued to report Conflicted: False and Programmed: True. Operators had no indication that a listener was being skipped. The Conflicted status condition is now correctly set on the resource whose listener was dropped due to a hostname collision.

TrafficExtension now enforced when waypoint is in a different namespace

Previously, a TrafficExtension defined in a service’s namespace was silently dropped when the waypoint serving that service lived in a different namespace. No error was returned and the waypoint processed traffic as if the extension were not present. TrafficExtension resources now work correctly in cross-namespace shared waypoint topologies.

Waypoint inbound routes now include all matching VirtualServices

A regression caused waypoints to include inbound routes only from the first VirtualService matching a given hostname. All other VirtualServices sharing that hostname were silently dropped, returning 404 NR for any path defined exclusively in those resources. Waypoints now correctly build the inbound VirtualHost from all matching VirtualServices.

🗑️ Removed features

🚧 Known issues

The Solo team fixes bugs, delivers new features, and makes changes on a regular basis as described in the changelog. Some issues, however, might impact many users for common use cases. These known issues are as follows:

Version-specific restrictions:

• In the Solo distribution of Istio 1.30 and 1.31, WorkloadEntry resources created by autoregistration might not carry the networking.istio.io/tunnel: http label. When that label is absent, ambient-enrolled pods send connections to the VM as plaintext instead of over HBONE. Because plaintext connections carry no mTLS client identity, any AuthorizationPolicy that uses source.principals to control inbound access to the VM always returns 403. As a workaround, manually label the gateway WorkloadEntry with networking.istio.io/tunnel: http to force HBONE, or replace source.principals conditions with source.ipBlocks until a patch release is available.

Other known Istio limitations:

• In-pod enrollment severs established connections (upstream issue istio/istio#52218): When the Istio CNI plugin enrolls a running pod into the ambient mesh, it inserts iptables rules into the pod’s network namespace while the pod is already serving traffic. Any TCP connections that were established before the rules were inserted are severed because the rules redirect existing sockets through ztunnel, which the remote side is not expecting. The following mitigations reduce the impact:

  • Enable detection retries in the CNI plugin so that it retries the ambient-membership lookup and holds the pod back before it begins serving, rather than letting it through un-enrolled: set ambient.enableAmbientDetectionRetry: true in your istiod Helm values.
  • Stagger pod startups and avoid burst-scaling a Deployment into an ambient-enabled namespace, so enrollment occurs before the pod begins serving.
  • Add istio.io/dataplane-mode: none to pods that cannot tolerate mid-life enrollment, and enroll them during a planned maintenance window.
  • Configure app-level retries so that clients automatically reconnect after a dropped connection.

• Multi-subnet cluster connection failures (upstream issue ztunnel#1456): When a workload in an ambient mesh connects to a multi-subnet cluster (such as an SQL Server Multi-Subnet Cluster), ztunnel’s TCP proxy optimistically completes the TCP handshake for all connection attempts before verifying the upstream connection. Depending on race conditions, the client can complete a TCP handshake with an inactive server instance. When the client then sends data, the connection fails because the upstream connection cannot be established. As a workaround, opt the affected workloads out of the mesh by adding the istio.io/dataplane-mode: none label to the pod or namespace. A fix is available in upstream Istio and will be included in a future Solo release. The fix introduces an ambient.istio.io/connect-strategy: FIRST_HEALTHY_RACE annotation on a ServiceEntry that instructs istiod to use a healthy-first connection strategy for the affected external service. See istio#59083 for details.