Skip to content
You are viewing the documentation for Solo Enterprise for Istio, formerly known as Gloo Mesh (OSS APIs).

Release notes

Page as Markdown

Review summaries of the main changes in the Solo Enterprise for Istio 1.30 release.

Make sure that you review the breaking changes 🔥 that were introduced in this release and the impact that they have on your current environment.

The release notes include important installation changes and known issues. They also highlight ways that you can take advantage of new features or enhancements to improve your product usage.

For more information, see the following related resources:

  • Ambient or sidecar mesh upgrade guides: Steps to upgrade your service meshes to the latest Istio version supported by Solo Enterprise for Istio.
  • Solo UI upgrade guide: Steps to upgrade the Solo UI to the latest supported version.
  • Version reference: Information about Solo’s version support.

🔥 Breaking changes

Review details about the following breaking changes. The severity is intended as a guide to help you assess how much attention to pay to this area during the upgrade, but can vary depending on your environment.

🚨 High

Review severe changes that can impact production and require manual intervention.

  • No high-severity changes are currently reported.

🔔 Medium

Review changes that might have impact to production and require manual intervention, but possibly not until the next version is released.

ECS discovery requires additional IAM permission

If you use ECS integration, istiod now requires permission to perform the ec2:DescribeNetworkInterfaces AWS action in addition to the existing List and Describe permissions on ECS clusters, services, and tasks. Update your IAM role policy to add this permission before you upgrade. Without it, ECS workload discovery fails after the upgrade.

ℹ️ Low

Review informational updates that you might want to implement but that are unlikely to materially impact production.

istio.io/ingress-use-waypoint moved from annotation to label on WorkloadEntries

In peered multicluster environments, the istio.io/ingress-use-waypoint value on auto-generated WorkloadEntry resources is now tracked as a label instead of an annotation. If you built tooling that reads this value from WorkloadEntry annotations, update it to read from WorkloadEntry labels instead.

The label now propagates through the peering layer as follows:

  • If a local Service exists in the cluster, its istio.io/ingress-use-waypoint label is authoritative and propagates to WorkloadEntry resources on remote clusters. Setting the label to false on the service overrides any namespace-level setting.
  • If no local Service label is set but the namespace has istio.io/ingress-use-waypoint: "true", the namespace label propagates to WorkloadEntry resources on remote clusters.
  • If no local Service exists, the label is aggregated from remote WorkloadEntry labels. If any remote has the label set to true, the generated ServiceEntry on the observing cluster inherits it.

platforms.ecs Helm value deprecated

The platforms.ecs Helm value is deprecated in favor of platforms.aws. The previous path is still supported for backwards compatibility, but will be removed in a future release. Update your Helm values to use the new structure.

Before:

platforms:
  ecs:
    accounts:
      - role: arn:aws:iam::1111111111:role/istiod
        domain: example.acme
        regions: ["us-east-1"]

After:

platforms:
  aws:
    ecsDiscovery: true
    accounts:
      - role: arn:aws:iam::1111111111:role/istiod
        domain: example.acme
        regions: ["us-east-1"]

🚧 New known issues

In the Solo distribution of Istio 1.30 and 1.31, WorkloadEntry resources created by autoregistration might not carry the networking.istio.io/tunnel: http label. When that label is absent, ambient-enrolled pods send connections to the VM as plaintext instead of over HBONE. Because plaintext connections carry no mTLS client identity, any AuthorizationPolicy that uses source.principals to control inbound access to the VM always returns 403. As a workaround, manually label the gateway WorkloadEntry with networking.istio.io/tunnel: http to force HBONE, or replace source.principals conditions with source.ipBlocks until a patch release is available.

Canary upgrades with the Gateway API cause brief traffic blackhole on tag flip

When Gateway API resources manage ingress traffic, flipping the default revision tag to a new istiod causes the old control plane to push empty xDS to Gateway pods that are still connected to it. Because the new istiod has not yet connected, Gateway pods have no routing rules for approximately 15–30 seconds, blackholing all Gateway-managed ingress traffic.

The upstream fix is available in Istio 1.30.2 and later. For workarounds, see Canary upgrades with the Gateway API.

🌟 New features

Review the following new features that are introduced in version 1.30 and that you can enable in your environment.

Additional service hostnames for segments

You can now define additional first-class DNS names for services in a segment using the spec.hostnames field on the Segment resource. Unlike spec.aliases, which stores additional hostnames as annotations, spec.hostnames adds the generated names directly as hosts to the generated ServiceEntry. DestinationRules, VirtualServices, and HTTPRoutes can target these hostnames natively without additional configuration.

Hostname patterns use the same URI Template RFC 6570 Level 1 syntax as spec.aliases, supporting variables such as {service}, {namespace}, {segmentDomain}, and label or annotation lookups. Patterns defined across clusters in the same segment are unioned, so each service is addressable by all patterns regardless of which cluster defined them.

For more information, see Additional service hostnames with spec.hostnames.

Agentgateway as a waypoint proxy (alpha)

You can now deploy Solo Enterprise for agentgateway as a waypoint proxy in your ambient mesh by using the enterprise-agentgateway-waypoint GatewayClass. Agentgateway integrates with Istio’s HBONE protocol and supports CEL-based Layer 7 policy enforcement via the EnterpriseAgentgatewayPolicy CRD. This feature is designed for AI and agentic workloads that require intelligent, policy-driven traffic management at L7.

Key capabilities include:

  • CEL-based authorization: Write flexible authorization policies using CEL expressions, including conditions on workload identity claims extracted from mTLS peer certificates.
  • AI provider backends: Route traffic to external AI providers such as OpenAI and Anthropic, with secret-based credential injection.
  • Workload identity propagation (WIMSE): Forward Workload Identity Tokens on outbound requests to carry source workload identity across trust boundaries.
  • Multicluster support: The agentgateway control plane runs in a single cluster, while waypoint pods can run in any enrolled cluster.

To get started, see Install agentgateway as a waypoint.

Canary deployments with Argo Rollouts documentation

New documentation is available for using Argo Rollouts to manage canary deployment strategies for workloads in your ambient mesh. The documentation covers two traffic management approaches:

  • Gateway API: Uses the Argo Rollouts Gateway API plugin to split traffic via HTTPRoute backendRef weights. Recommended for ambient mesh environments.
  • VirtualService and DestinationRule subsets: Uses Argo’s Istio API support to split traffic via destination rule subset labels and virtual service route weights. Works for both sidecar and ambient datapaths.

Both approaches integrate with Solo Enterprise for Istio’s global services feature to manage traffic splits across local service FQDNs, global hostnames, and cross-cluster destinations. Multicluster routing considerations and flat network patterns are also covered.

To get started, see Canary deployments with Argo Rollouts.

CEL and workload claims authorization for L4 traffic (alpha)

You can now use Common Expression Language (CEL) expressions in ztunnel AuthorizationPolicy rules to enforce fine-grained L4 access control based on workload identity claims. Enable workload claims in ztunnel (ENABLE_WORKLOAD_CLAIMS=true) to allow istiod to embed pod-level identity claims directly in each workload certificate. Ztunnel reads these claims at connection setup time and exposes them in a source.claims map keyed by fully-qualified dotted names, such as source.claims['istio.io.workload.namespace']. You can also attach custom security claims to workloads using pod annotations prefixed with solo.io.security-claims/, which are accessible as source.claims['solo.io.security-claims.<key>'].

This feature is available as an alpha feature in version 1.30 and later of the Solo distribution of Istio. For more information, see CEL and workload claims authorization for L4 traffic.

Certificate revocation list (CRL) enforcement (alpha)

Ztunnel now enforces Certificate Revocation Lists on all new connections, both inbound and outbound, when you use a plugged-in CA. Add a ca-crl.pem entry to the cacerts secret in istio-system, enable PILOT_ENABLE_CA_CRL=true on istiod, and set peerCaCrl.enabled=true on the ztunnel Helm chart. To enforce CRL on east-west gateway connections, set eastwest.peerCaCrl.enabled=true on the peering Helm chart. Istiod propagates the CRL as a ConfigMap to every namespace, where ztunnel mounts and enforces it. CRL updates are reloaded automatically within the kubelet’s sync interval (typically 60–90 seconds) with no ztunnel restart required.

Enforcement applies to all new connections. Existing in-flight connections are not affected by CRL updates and remain active until ztunnel restarts.

For more information, see Certificate revocation list enforcement.

EC2 ambient mesh integration (alpha)

You can now onboard workloads running on Amazon EC2 instances to your Istio ambient mesh. The new istioctl ec2 add-workload command in the Solo distribution of istioctl bootstraps a ztunnel binary on the EC2 instance, using the instance’s IAM instance profile and the EC2 Instance Metadata Service (IMDS) for authentication. EC2 workloads join the mesh with full mTLS support, receive their own SPIFFE identity, and can be managed with Istio authorization policies the same as in-cluster pods.

Key capabilities include:

  • Auto-discovery: Istiod discovers EC2 instances tagged with ec2.solo.io/* metadata and automatically creates the WorkloadEntry, Service, and gateway WorkloadEntry resources.
  • Multi-account and multi-region support: Configure istiod to discover EC2 instances across multiple AWS accounts and regions.
  • Multi-workload instances: A single EC2 instance can host multiple workloads, each with its own SPIFFE identity, using ztunnel’s shared_vm proxy mode.
  • Public and private IP support: Use the instance’s private IP by default, or configure public IP routing for instances outside the EKS VPC.

To get started, see Add EC2 instances to the mesh.

Istio 1.30 support

You can now run Solo Enterprise for Istio with Istio 1.30. For more information, see the version support matrix, and the Solo distribution of Istio changelog for 1.30.

istioctl experimental jwks command

Istiod now exposes a JWKS debug endpoint that publishes the CA’s signing certificate in JSON Web Key Set format. Use the new istioctl experimental jwks command in the Solo distribution of istioctl to retrieve the JWKS document from a running istiod instance. This document is useful for debugging certificate-related issues and verifying the trust anchor used for workload authentication. For more information, see istioctl experimental jwks.

L4 ztunnel-native egress (alpha)

Version 1.30 introduces L4 ztunnel-native egress as an alpha feature. This approach enforces identity-based egress policy directly in the existing ztunnel DaemonSet, with no dedicated proxy pods deployed. Traffic exits from the ztunnel pod’s network namespace, which allows you to use Kubernetes NetworkPolicy to ensure workload pods cannot reach external IPs directly. Use L4 ztunnel-native egress for TCP, TLS, and HTTP traffic where source identity or destination port and address are sufficient for policy enforcement. For HTTP traffic that requires method, path, or header-level enforcement, use L7 waypoint egress instead.

Version 1.30 also introduces the solo.io/sidecar-skip-waypoint: "true" annotation for ServiceEntry resources. In clusters where sidecar-injected workloads and ambient workloads coexist, this annotation tells sidecars to ignore the istio.io/use-waypoint binding on a ServiceEntry and continue using their existing registry-based routing. Ambient workloads still honor the binding and route through the ztunnel-egress Gateway. Add this annotation to any ServiceEntry that uses the ztunnel-egress Gateway in a mixed-mode cluster.

To get started, see L4 ztunnel-native egress.

License keys from mounted volumes

You can now load license keys from a mounted volume instead of a Kubernetes Secret. Provide the path to the mounted volume using the license.volume Helm value. This capability is useful in environments where secrets management tooling writes credentials to a mounted file path rather than a Kubernetes Secret. For more information, see Licensing.

Mesh configuration validation metrics

Istiod now exposes two new Prometheus gauges to track the validity of its configuration:

  • pilot_meshconfig_validation_status: Returns 0 when the current mesh configuration is valid and 1 when it is invalid.
  • pilot_meshnetworks_validation_status: Returns 0 when the current mesh networks configuration is valid and 1 when it is invalid.

Use these metrics to alert on configuration parse or validation errors and to confirm that configuration updates have been applied correctly. For more information, see Metrics.

NodePort status condition for east-west gateways

Istiod now sets a gloo.solo.io/NodePortConfigured status condition on east-west gateways configured for NodePort peering. When the gateway’s managed Service does not have a port named hbone, the condition is set to False with reason MissingHbonePort. When a valid HBONE port is present, the condition is True with reason Programmed. Use this condition to verify NodePort peering gateway health without inspecting the backing Service directly. For more information, see Multicluster peering.

Peering label controls

Version 1.30 introduces three new environment variables for controlling how labels and annotations flow through the ServiceEntry and WorkloadEntry resources that Istiod auto-generates during multicluster ambient mesh peering.

  • PEERING_EXCLUDED_LABELS: A comma-delimited list of label keys to exclude from auto-generated peering resources’ metadata.
  • PEERING_EXCLUDED_ANNOTATIONS: A comma-delimited list of annotation keys to exclude from auto-generated peering resources’ metadata.
  • PILOT_PEERING_WE_EXCLUSION_LABELS: A comma-separated list of label keys. Any WorkloadEntry that carries at least one of these keys is not selected by a peering-generated ServiceEntry. Non-peering ServiceEntry resources are unaffected. Defaults to gloo.solo.io/parent_name, which prevents VirtualDestination-generated WorkloadEntry resources from being selected by peering ServiceEntry resources.

Use PEERING_EXCLUDED_LABELS and PEERING_EXCLUDED_ANNOTATIONS to prevent internal or environment-specific metadata from propagating across clusters. Use PILOT_PEERING_WE_EXCLUSION_LABELS to prevent peering ServiceEntry resources from accidentally selecting WorkloadEntry resources managed by other components.

Policy inheritance for global services (alpha)

Non-routing policy resources that target a Kubernetes Service can now automatically extend to the corresponding global service as an alpha feature. The following types support inheritance: AuthorizationPolicy, Telemetry, EnvoyFilter, TrafficExtension, WasmPlugin, and RequestAuthentication. You can enable inheritance per-resource by adding the solo.io/service-inheritance: "true" annotation to a policy resource, or globally for all eligible resources by setting the ENABLE_SERVICE_INHERITANCE=true environment variable on the istiod deployment.

For more information, see Policy attachment for global services.

Route to global services with GlobalService parentRef (alpha)

Route types such as HTTPRoute, GRPCRoute, TCPRoute, and TLSRoute can now attach routing rules directly to a global service as an alpha feature. Use kind: GlobalService (group: networking.istio.io) as the parentRef to attach a route to the global service. This allows you to apply routing behavior such as retries, timeouts, or header-based matching specifically to globally routed traffic, separately from your local service routing rules.

For more information, see Route to global services.

Solo UI (alpha)

Version 1.30 introduces the Solo UI, a new management experience for Solo Enterprise for Istio. The Solo UI deploys alongside your Istio ambient mesh in both single- and multicluster environments, and provides a visual, telemetry-driven interface for observing and managing your ambient mesh.

Key capabilities include:

  • Visual service graph: Review service-to-service communication, applied policies, and mTLS status across your mesh.
  • Live metrics: View workload-level metrics such as request rates, error rates, and latency, powered by an integrated ClickHouse data store and OpenTelemetry pipeline.
  • Global services: In multicluster setups, review all services exposed across the multicluster mesh and their health statuses.
  • Resource overview: Browse all Istio and Kubernetes resources across registered clusters in one place.

You can install the Solo UI by using the solo-enterprise/management and solo-enterprise/relay Helm charts in the solo-enterprise namespace, separately from the existing gloo-platform management components. The existing gloo-platform-based management system continues to be supported in version 1.30, and you can run both side by side during evaluation. However, the new Solo UI experience is strongly recommended to support new mesh-based observability features.

To get started, see Install the Solo UI. For a description of the components and ports, see Architecture. For information about the telemetry pipeline and data store, see ClickHouse data store. Note that this feature requires an Enterprise-level license.

VM ambient mesh integration enhancements

Version 1.30 introduces two enhancements to VM onboarding in ambient mesh.

Simplified onboarding with istioctl vm add-workload: The new istioctl vm add-workload command in the Solo distribution of istioctl automates the steps required to onboard a generic VM to the ambient mesh. The command creates the Kubernetes namespace, ServiceAccount, ServiceEntry, and WorkloadEntry for the VM workload, and outputs the bootstrap token and Docker command needed to start ztunnel on the VM.

Multi-workload VM support: A single VM can now host multiple workloads, each with its own SPIFFE identity, certificates, and authorization policy enforcement. Ztunnel runs on the VM in shared_vm proxy mode, which combines gateway, inbound, and outbound capabilities in a single ztunnel instance. Applications use SOCKS5 with per-identity username selection for outbound requests, and inbound traffic is delivered via a double-HBONE flow through the VM gateway.

To get started, see Add VMs to the mesh.

🔄 Feature changes

Review the following changes that might impact how you use certain features in your environment.

Gloo platform management plane

The existing gloo-platform-based management system and Gloo UI continue to be supported in version 1.30, such as for use with sidecar meshes and Basic- or Premium-level licenses.

However, for ambient meshes and Enterprise-level license, the new Solo UI experience is strongly recommended to support new mesh-based observability features. Consider moving the new UI for an improved and streamlined observability system. Note that you can run both UI systems side-by-side during evaluation.

Enhanced istioctl multicluster check validation

The istioctl multicluster check command has been significantly expanded in version 1.30 with new checks and output capabilities.

New checks:

  • Missing local Segment resources: Detects local Segment resources whose absence causes auto-generated ServiceEntry resources from a peered cluster to be silently dropped, and reports rejected peer segments with an actionable error message.
  • Rejected peer Segment states: Detects rejected peer Segment states caused by hostname mismatches and overlapping Segment domains.
  • CNI DNS capture: Verifies that AMBIENT_DNS_CAPTURE is set to true in the istio-cni-config ConfigMap. DNS capture is required for global hostname lookups (*.mesh.internal) to resolve correctly across clusters. Returns a warning if the value is missing or disabled.
  • Remote gateway cluster label: Verifies that each istio-remote Gateway resource has the topology.istio.io/cluster label set to the ID of the remote cluster it represents. Returns an error if the label is missing, and an informational message if the label points to a cluster not included in the --contexts flag.
  • ISTIO_META_NETWORK environment variable: Validates that all Istio proxy containers and sidecar-injector ConfigMap templates have ISTIO_META_NETWORK set to match the network label on the istio-system namespace.
  • Remote gateway service account: Checks that a remote gateway’s effective service account matches the corresponding peer east-west gateway.
  • Hostname conflicts: Reports hostname conflicts by reading conflict annotations on auto-generated ServiceEntry resources.
  • NodePort east-west gateway health: Detects NodePort east-west gateways that have fallen back to reporting a ClusterIP address, and validates that the backing Service has the required HBONE port.

New capabilities:

  • Structured output: Pass -o json or -o yaml to emit all check results as a structured document grouped by cluster.
  • Offline analysis: Run against extracted bug-report directories for multicluster analysis without direct cluster access.
  • Dynamic gateway discovery: East-west gateway names are discovered dynamically from Gateway resources instead of being hardcoded to istio-eastwest.

For more information, see the istioctl multicluster check reference.

istioctl bootstrap --revision flag

The istioctl bootstrap command now accepts a --revision (-r) flag for targeting a specific control plane revision. When set, the command looks up istiod-{revision} instead of the default istiod service. Use this flag in environments with multiple control plane revisions running simultaneously.

istioctl ztunnel-config connections output

The istioctl ztunnel-config connections command now includes a CONN ID column that displays the connection ID for each inbound and outbound ztunnel connection, to aid in debugging connection-level issues.

Public IP support for ECS tasks

ECS tasks can now use their public IP address for mesh routing. This setup is useful when ECS tasks run in a subnet or VPC that is not directly routable from your EKS cluster. To enable public IP routing, add the ecs.solo.io/use-public-ip=true tag to both the ECS service and task. If the tag is set but the task has no public IP assigned, the WorkloadEntry is annotated with solo.io/endpoint-status=unhealthy and traffic is not routed to the task until the address is resolved.

For more information, see Add ECS services to the mesh.

🐛 Bug fixes

Cross-cluster sidecar mTLS enforcement in flat-network peering

Cross-cluster traffic from a sidecar to a non-HBONE sidecar in a flat-network peering topology is now sent as Istio mTLS instead of plaintext.

Cross-cluster subset routing with custom labels

In multicluster ambient meshes, custom pod and service labels are now propagated to autogenerated WorkloadEntry objects created by the peering controller in connected clusters. Previously, these WorkloadEntries were created with only a minimal set of labels (app, solo.io/*, security.istio.io/tlsMode), which prevented DestinationRule subset matching across clusters and caused cross-cluster requests routed via subsets to fail with 503: no_healthy_upstream. This fix is required for Argo Rollouts canary deployments in multicluster environments that use DestinationRule subsets for traffic splitting.

This fix applies to both standard (non-flat) and flat network peering topologies. In non-flat environments, service-level labels are propagated to the aggregate autogenerated WorkloadEntry per service. In flat network environments, both service-level and pod-level labels are propagated to the per-pod autogenerated WorkloadEntries, enabling finer-grained subset matching such as rollouts-pod-template-hash or custom canary labels.

This fix is feature-gated. To enable it, set the ENABLE_PEERING_LABEL_PROPAGATION pilot environment variable to one of the following values:

  • all: Propagate all labels from the source pod or service.
  • A comma-delimited list of label keys: Propagate only the specified labels, which reduces data over the wire. For example, app,version,canary.

For more information about cross-cluster canary deployments, see Canary deployments with Argo Rollouts.

ECS discovery

Several ECS discovery issues are fixed:

  • Long hostname label: Services with long cluster or service names could not join the mesh because the ecs.solo.io/hostname label value exceeded the 63-character Kubernetes limit. The label value is now truncated to fit.
  • Missing domain configuration: Workloads failed to register as WorkloadEntry resources when no domain was specified for an AWS account. A default domain is now used so workloads are always reachable within the mesh.
  • Custom system namespace: The ECS controller did not respect custom system namespace configurations and defaulted to istio-system. The controller now uses the configured namespace.
  • AWS tag filtering: AWS resource tags are now filtered before being applied as Kubernetes labels. Only tags with *.istio.io/, *.solo.io/, or *.kubernetes.io/ domain prefixes are passed through to generated Kubernetes objects.

Flat-network peering with named targetPort silently drops remote endpoints

In flat-network peering topologies, services that use a named targetPort field no longer cause remote endpoints to be silently dropped.

Gateway annotation updates trigger unnecessary restarts

Adding certain annotations to gateway resources no longer triggers unnecessary pod restarts. The following cases are fixed:

  • Adding the draining annotation to an east-west Gateway.
  • Adding the traffic distribution annotation to a waypoint Gateway.
  • Adding a service-type annotation to an east-west Gateway.

Global waypoint label exceeds Kubernetes 63-byte limit

An issue where the global waypoint label value could exceed the Kubernetes 63-byte limit when the waypoint name, namespace, and domain were long is fixed. Previously, this caused the label to be rejected.

Named targetPort routing for global services

Traffic routing failures for global services that use named targetPort values are fixed. Previously, requests could be sent to the wrong port when routing across clusters or to local services exposed via named ports.

Peering controller skips unenrolled namespace services

The peering controller no longer creates ServiceEntry resources for services in namespaces that are not enrolled in the mesh. Previously, Service resources with global scope labels in namespaces without ambient or sidecar injection were incorrectly included in peering.

Remote peering ServiceEntry missing for waypoint-selecting services

In 1.30.0 and 1.30.1, the peering controller did not create a remote ServiceEntry for services whose pod selector matched waypoint proxy pods. Any service with a global or segment scope now has a peered ServiceEntry created. This fix is included in 1.30.2.

prefer-other ServiceEntry routing and DNS

Two prefer-other ServiceEntry resolution issues are fixed:

  • Non-deterministic routing: Traffic could be non-deterministically routed through a prefer-other ServiceEntry instead of a non-prefer-other ServiceEntry with the same hostname. This issue occurred when a VirtualService referenced a hostname served by both types and an explicit Sidecar scope limited the visibility of the non-prefer-other ServiceEntry.
  • DNS flip-flopping: DNS responses for a hostname served by both a prefer-other and a non-prefer-other ServiceEntry could include addresses from both, causing inconsistent DNS resolution.

Segment domain and aliases updates regenerate ServiceEntry hostnames

Updating the domain or aliases fields on a remote cluster’s Segment resource now correctly regenerates peered ServiceEntry hostnames. Previously, changes to these fields did not trigger hostname regeneration.

Service takeover default traffic distribution

Service resources with takeover enabled via solo.io/service-takeover=true or solo.io/service-scope=global-only now default to PreferNetwork traffic distribution when no other distribution is set. Previously, the takeover Service traffic distribution was inconsistent with the federated Service distribution, which defaults to PreferNetwork or inherits the base Service setting.

Sidecar generation with service takeover and no local service

Sidecar generation no longer fails when service takeover is enabled and no local Service exists in the cluster.

Stale metadata-exchange configuration for outbound clusters

Outbound clusters are no longer served stale metadata-exchange configuration when the set of service endpoints supporting HBONE changes.

Traffic distribution precedence for peered ServiceEntries

The networking.istio.io/traffic-distribution annotation on auto-generated peered ServiceEntry resources is now calculated using the correct precedence order. Previously, namespace-level annotations were ignored and spec.trafficDistribution could be overridden incorrectly by the service annotation.

The annotation now follows this precedence order, from highest to lowest:

  1. spec.trafficDistribution field on the Kubernetes Service
  2. networking.istio.io/traffic-distribution annotation on the Service
  3. networking.istio.io/traffic-distribution annotation on the namespace
  4. Default: PreferNetwork

Namespace annotations are now respected as a cluster-wide or namespace-wide default for all peered services. If the namespace annotation changes, all services in that namespace are automatically reprocessed.

🚧 Known issues

The Solo team fixes bugs, delivers new features, and makes changes on a regular basis as described in the changelog. Some issues, however, might impact many users for common use cases. These known issues are as follows:

Version-specific restrictions:

  • To use the Solo distribution of Istio 1.30 for multicluster setups in which waypoint proxies are deployed, use patch version 1.30.2 or later. Patch versions 1.30.0 and 1.30.1 contain a bug in which the peering controller does not create a remote `ServiceEntry` for services whose pod selector matches waypoint proxy pods.
  • In the Solo distribution of Istio 1.30 and 1.31, WorkloadEntry resources created by autoregistration might not carry the networking.istio.io/tunnel: http label. When that label is absent, ambient-enrolled pods send connections to the VM as plaintext instead of over HBONE. Because plaintext connections carry no mTLS client identity, any AuthorizationPolicy that uses source.principals to control inbound access to the VM always returns 403. As a workaround, manually label the gateway WorkloadEntry with networking.istio.io/tunnel: http to force HBONE, or replace source.principals conditions with source.ipBlocks until a patch release is available.

Other known Istio limitations:

  • In-pod enrollment severs established connections (upstream issue istio/istio#52218): When the Istio CNI plugin enrolls a running pod into the ambient mesh, it inserts iptables rules into the pod’s network namespace while the pod is already serving traffic. Any TCP connections that were established before the rules were inserted are severed because the rules redirect existing sockets through ztunnel, which the remote side is not expecting. The following mitigations reduce the impact:

    • Enable detection retries in the CNI plugin so that it retries the ambient-membership lookup and holds the pod back before it begins serving, rather than letting it through un-enrolled: set ambient.enableAmbientDetectionRetry: true in your istiod Helm values.
    • Stagger pod startups and avoid burst-scaling a Deployment into an ambient-enabled namespace, so enrollment occurs before the pod begins serving.
    • Add istio.io/dataplane-mode: none to pods that cannot tolerate mid-life enrollment, and enroll them during a planned maintenance window.
    • Configure app-level retries so that clients automatically reconnect after a dropped connection.
  • Multi-subnet cluster connection failures (upstream issue ztunnel#1456): When a workload in an ambient mesh connects to a multi-subnet cluster (such as an SQL Server Multi-Subnet Cluster), ztunnel’s TCP proxy optimistically completes the TCP handshake for all connection attempts before verifying the upstream connection. Depending on race conditions, the client can complete a TCP handshake with an inactive server instance. When the client then sends data, the connection fails because the upstream connection cannot be established. As a workaround, opt the affected workloads out of the mesh by adding the istio.io/dataplane-mode: none label to the pod or namespace. A fix is available in upstream Istio and will be included in a future Solo release. The fix introduces an ambient.istio.io/connect-strategy: FIRST_HEALTHY_RACE annotation on a ServiceEntry that instructs istiod to use a healthy-first connection strategy for the affected external service. See istio#59083 for details.