This release note describes what’s different between Solo builds of Istio versions 1.28.0 and 1.29.0.

Upgrade Notes

Manual configuration of ECS platform discovery

If you install Istio using Helm, Gloo Operator, or istioctl, no action is required.

If you manually configure ECS platform discovery by setting the ECS_ACCOUNTS environment variable, you will need to add an array of regions for each account. For example:

  # old
ECS_ACCOUNTS="example.acme,arn:aws:iam::1111111111:role/istiod"

# new
ECS_ACCOUNTS="example.acme,arn:aws:iam::1111111111:role/istiod,[us-east-1 us-east-2]"

If you wish to continue using the default region of the associated AWS role, simply use the empty array []. For example:

  ECS_ACCOUNTS="example1.acme,arn:aws:iam::1111111111:role/istiod,[]"

General Changes

Solo Flavor Changes

  • Added aliases to the Segment custom resource, enabling a high degree of customization of hostnames for global services. Aliases allow peered multi-cluster services to be addressed by user-defined names, providing flexibility in how services are discovered and routed across clusters.

  • Improved the istioctl multicluster check command with the following enhancements:

    • Show gateway addresses, peer cluster addresses, and globally shared services
    • Skip the stale workload check unless flat-network is detected
    • Network configuration validation
    • Validation for the compatibility of intermediate certificates between peered clusters
    • Enhanced validation checks for flat-network configurations

  • Added several enhancements to istioctl ztunnel-config:

    • New endpoints subcommand (aliases: ep, endpoint) to retrieve endpoint information for a specific service, accepting --service, --hostname, and --service-namespace flags with table, JSON, and YAML output
    • NETWORK and NETWORK GATEWAY columns to the workloads subcommand output for better visibility into workload network configuration
    • Support to retrieve connections from an east-west gateway via the connections subcommand

  • Added new CLI options to the istioctl multicluster expose and link commands:

    • --data-plane-service-type on expose: specifies a comma-delimited list of data plane service types peers can use (loadbalancer or nodeport, default loadbalancer)
    • --preferred-data-plane-service-type on link: specifies the preferred data plane service type when peering with a remote cluster (loadbalancer or nodeport, default loadbalancer)
    • --generate on link: generates Gateway manifests for multiple contexts at once

  • Added ambient and ztunnel enhancements:

    • Ztunnel Helm support for dnsPolicy and dnsConfig customization
    • A mesh-wide escape hatch based on port matching for outbound traffic impacted by ztunnel capture, configured via AMBIENT_EXCLUDE_OUTBOUND_PORTS environment variable (e.g., AMBIENT_EXCLUDE_OUTBOUND_PORTS="1443,16000-16010")
    • Native nftables support for the port exclusion escape hatch (AMBIENT_EXCLUDE_OUTBOUND_PORTS now works with both iptables and nftables backends)

  • Added several east-west gateway improvements:

    • Support for breaking connections to pods that are terminated
    • PodDisruptionBudget and HorizontalPodAutoscaler resources to the kube-eastwest gateway template, customizable via the gateway class ConfigMap

  • Added a new peering Helm chart for managing istio-eastwest and istio-remote gateways. This chart provides an alternative to using istioctl multicluster expose and istioctl multicluster link commands, supporting declarative management of east-west and remote peering gateways via Helm.

  • Added peering metrics to monitor the health and performance of peer cluster connections:

    • peer_connection_state: Gauge metric tracking the connection state of peer clusters (1 = connected, 0 = disconnected), labeled by peer and source cluster.
    • peer_xds_config_size_bytes: Distribution metric tracking the size of XDS configuration received from peer clusters, labeled by peer, source, and type.
    • peer_convergence_time: Distribution metric tracking the time from sending an XDS request to a peer until receiving a response, labeled by peer, source, type, and success.

  • Added the solo.io/prefer-other ServiceEntry annotation to enable preference-based ServiceEntry conflict resolution across namespaces. When multiple ServiceEntry resources define the same hostname, this annotation indicates that Istio should prefer any ServiceEntry without the annotation, regardless of namespace. This allows for better behavior in migration scenarios where legacy ServiceEntry resources can be marked as fallbacks while new ServiceEntry resources take priority.

  • Added namespace-level traffic distribution annotation. Services inherit traffic distribution from namespace annotation when not explicitly set on the service.

  • Added multi-region support to ECS platform discovery. Previously only ECS resources within the default region of the configured AWS role would be discovered.

  • Added the ability to specify authorized namespaces for debug endpoints when ENABLE_DEBUG_ENDPOINT_AUTH=true. Enable by setting DEBUG_ENDPOINT_AUTH_ALLOWED_NAMESPACES to a comma-separated list of authorized namespaces. The system namespace (typically istio-system) is always authorized.

  • Added support to retry failed remote peer creation or updates when PEERING_AUTOMATIC_LOCAL_GATEWAY is enabled.

  • Removed the explicit requirement for specifying an HBONE listener on an istio-remote gateway resource. If an istio-remote gateway’s preferred data plane service type is loadbalancer, a network gateway is created with the well-known HBONE port value 15008. Backwards compatibility is maintained by still allowing an HBONE listener to be specified.

  • Improved istioctl command help, refreshing CLI descriptions and examples with clearer guidance for bootstrap, ecs service-add, and multicluster’s check, expose, and link commands.

  • Fixed multiple NodePort peering issues:

    • ServiceInfo events not propagated after the HboneNodePort field changed, which broke NodePort peering when a gateway was annotated after its initial creation
    • Nodes in a NotReady state still being used for multi-cluster traffic routing
    • Node IP address type selection now only considers internal IP addresses
    • A comma-delimited value for the peering.solo.io/data-plane-service-type annotation preventing node workloads from being sent to peers
    • Simultaneous processing of node WorkloadEntry and gateway-derived ServiceEntry delete events causing the node WorkloadEntry to persist
    • Generated ServiceEntry and node WorkloadEntry not being cleaned up when the istio-remote gateway’s peering.solo.io/preferred-data-plane-service-type annotation was no longer set to NodePort
    • Node WorkloadEntry resources failing to be created when NodePort peering was enabled after the peer initially connected via LoadBalancer for an extended period of time

  • Fixed several waypoint interop issues:

    • Traffic skipping local waypoints when there were no healthy endpoints (traffic will now fail)
    • An istiod crash when attempting to use Waypoint interop with sidecar and gateway proxies
    • Ingress gateways applying the waypoint’s DestinationRule instead of the service’s own DestinationRule regardless of whether the istio.io/ingress-use-waypoint label was set
    • ingress-use-waypoint label propagation from namespace to federated service

  • Fixed various istioctl command issues:

    • The bootstrap and ecs add-service commands now respect the -i istio system namespace flag
    • The ecs add-service command no longer incorrectly reports that a task definition does not have an executionRoleArn defined

  • Fixed flat-network multi-cluster peering issues:

    • User WorkloadEntry-based endpoints not being properly peered across clusters, preventing VMs and external workloads from being reachable from proxies in different clusters
    • Traffic being sent to pods peered for flat-network multi-cluster which are Not Ready or Terminating

  • Fixed an issue where services with takeover enabled (solo.io/service-takeover=true or solo.io/service-scope=global-only) did not default their traffic distribution to PreferNetwork. This makes takeover service traffic distribution consistent with federated service behavior.

  • Fixed east-west gateway TLS listeners showing an incorrect “UnsupportedProtocol” status when PILOT_ENABLE_ALPHA_GATEWAY_API was disabled.

  • Fixed an issue with upgrading from 1.27 that created WorkloadEntry resources for Pods in remote clusters that are not reachable on the same network.

  • Fixed an issue causing Envoy clusters for peered global services to not have the tlsMode-istio transport socket configured.

  • Fixed an issue where istiod would generate invalid WorkloadEntry when remote services have unnamed ports.

  • Fixed an issue where istiod would refuse to peer with clusters when the remote cluster tries to declare a Segment’s domain as one that overlaps with another Segment’s domain.

  • Fixed an issue where status on Segment resources was not updating.

  • Fixed an issue where the remote peer did not have its address updated when PEERING_AUTOMATIC_LOCAL_GATEWAY is enabled.

  • Fixed an issue where updating the address in the istio-remote Gateway resource would not prompt the control plane to connect to the new address.

  • Fixed an issue that occurred when draining was enabled and no cluster name was found. A warning is now issued.

  • Fixed an issue where deleting an active ECS ServiceEntry or WorkloadEntry caused it to be recreated.

FIPS Flavor Changes

No changes in this section.