Overview

Use the Solo.io distribution of Istio in ambient mode to peer multiple clusters without relying on remote Kubernetes API access. Each cluster runs its own istiod, east-west gateway, and remote peer gateway. Peering connects the control planes through xDS (Discovery Service) over mutual TLS (mTLS), and the data plane uses HBONE (HTTP-Based Overlay Network Environment) through the same east-west gateways for service-to-service traffic.

The following diagram shows the control plane and data plane paths between two peered clusters. The same pattern applies as you add more clusters. For more information about the components in this ambient mesh setup, see the ambient mesh Architecture.

Figure: Multicluster ambient mesh architecture
Figure: Multicluster ambient mesh architecture

Control plane peering

In each cluster, you create an east-west gateway that exposes port 15012 for xDS and port 15008 for HBONE traffic. Peering uses istio-remote Gateway resources to represent the east-west gateway address of the remote cluster. The istiod instance in each cluster establishes an mTLS-secured xDS connection to the remote istiod by targeting the remote cluster’s east-west gateway address. In other words, istiod uses the address that the istio-remote gateway represents.

After the xDS connection is established, istiod exchanges federated service and workload information with the remote control plane. This model avoids the need for istiod to watch remote Kubernetes APIs, and it keeps service discovery exchange inside the mesh control plane.

Istiod serves configuration over xDS to ztunnel and other proxies. For ambient mode, ztunnel uses the xDS transport protocol but consumes custom resources that are optimized for L4 traffic, such as Address and Authorization. This approach keeps control plane updates focused on the service and workload data needed for ambient routing, and it avoids the overhead of Envoy-style xDS resources for ztunnel.

Figure: Multicluster control plane peering connection.
Figure: Multicluster control plane peering connection.

Data plane traffic

In ambient mode, ztunnel handles L4 traffic for workloads on each node. For cross-cluster traffic, the client-side ztunnel socket routes the request over HBONE to the remote cluster’s east-west gateway on port 15008. The east-west gateway then forwards the request to the target destination’s ztunnel socket in the remote cluster.

Because the east-west gateway acts as a ztunnel, it does not alter requests. Use waypoint proxies when you need L7 processing or policy enforcement in the request path. For more information, see Control in-mesh traffic with east-west gateways and waypoints.

HBONE is an HTTP CONNECT tunnel over mTLS on port 15008. Ztunnel uses the destination IP address and mesh identity to select the correct certificate and route the connection. In ambient mode, ztunnel multiplexes multiple application connections over pooled HBONE connections when the source identity, destination identity, and destination IP match. This keeps the data plane lightweight while preserving zero-trust semantics.

Ztunnel is intentionally minimal. It focuses on L4 security and connectivity, and it delegates L7 features such as retries, header-based routing, or external auth to waypoint proxies. This separation is a core reason the ambient data plane scales without a sidecar on every pod.

Figure: Multicluster routing, in which workload 1 in cluster 1 sends a request to workload 2 in cluster 2.
Figure: Multicluster routing, in which workload 1 in cluster 1 sends a request to workload 2 in cluster 2.

Default network vs flat network setups

Istio supports two types of network setups: traditional, or default setups, in which you use different networks for each cluster that are part of the multicluster mesh; and flat network, or single network setups, in which all your clusters use the same network.

Use the default network setup when each cluster runs in a distinct network. Cross-cluster traffic flows through east-west gateways, which provide stable addresses for xDS and HBONE traffic. For setup steps, see Default setup.

Use a flat network setup when all clusters share one network and workloads can reach each other by service IP address directly. With a flat network, data plane traffic does not need to traverse the east-west gateway, but you still create east-west and peering gateways so that istiod instances can connect over xDS. Flat network setups require a shared network name in the topology.istio.io/network label and global.network setting across clusters. However, each cluster must use unique pod and service CIDR ranges to avoid IP conflicts.

Flat networking is an advanced setup. Use it only if your environment already runs a flat network topology. For setup steps, see Flat networking (advanced).

Default networks: LoadBalancer vs NodePort for east-west and peering gateways

When you use the default network setup, you can optionally choose between using LoadBalancer services (default) for the east-west and peering gateways, or NodePorts so that cross-cluster traffic resolves to NodePort addresses.

Use the LoadBalancer method when you want a standard setup. Cross-cluster traffic resolves to the east-west gateway LoadBalancer address.

Use the NodePort method in the Solo distribution of Istio 1.28 and later when you want to avoid LoadBalancer services and reduce cross-cluster traffic costs. With NodePorts, data plane traffic resolves to NodePort addresses, but xDS still requires stable gateway addresses for control plane connections.

For more information about and considerations for using each method, see Best practices for multicluster peering. For setup steps, see the multicluster setup guides.

Performance and scale considerations

Peering scales efficiently because istiod does not rely on remote Kubernetes API watches. Instead, each istiod uses xDS to exchange only the federated service data that the control plane needs.

Scale limits

Ambient multicluster peering scales to extremely large deployments. In testing, Solo Enterprise for Istio successfully handles:

  • 100 million pods across 2,000 connected clusters
  • 2.5 million pods across 1,000 connected clusters
  • 2,000 pod changes per second across the mesh

These scales represent orders of magnitude beyond typical production environments. For further discussion on scaling, see The 100 Million Pod Mesh and Scale beyond 1 Million Pods: Introducing Gloo Mesh Ambient Multicluster on the Solo.io blog.

Resource efficiency

Ambient multicluster maintains low resource utilization even at extreme scales:

  • Control plane CPU: The control plane typically uses 10-30% of a single CPU core per cluster, even with millions of pods connected.
  • Control plane memory: The control plane has a minimal memory footprint, fitting within a Raspberry Pi-sized footprint.
  • Propagation time: Changes propagate across the mesh in less than 100ms, even at the largest scales.

Cluster sizing recommendations

Based on tested configurations, you can size your clusters as follows:

  • Services per cluster: 50-1,000 services
  • Pods per cluster: 2,500-50,000 pods
  • Nodes per cluster: 50-1,000 nodes
  • Istiod replicas: A single istiod replica per cluster is sufficient for most deployments, though you can scale up if needed
  • Service export: Export 10% or fewer of your services for global consumption across clusters

Configuration

No manual tuning or configuration scoping is required. Solo Enterprise for Istio ambient multicluster works out of the box with default Istio configuration, so that you can operate the mesh at any scale without additional configuration.

Differences from upstream multicluster

Upstream multicluster peering in the community Istio project uses remote secrets that contain kubeconfigs, which lets istiod watch remote Kubernetes APIs. The Solo distribution of Istio uses a decentralized, push-based peering model with istio-remote gateways and xDS between istiod instances. In addition, the Solo distribution supports data plane routing through either LoadBalancer or NodePort addresses for the east-west gateway, while still requiring a stable address for xDS connectivity.

If you previously used the multicluster feature in community Istio, and want to now migrate to multicluster peering in the Solo distribution of Istio, the DISABLE_LEGACY_MULTICLUSTER environment variable is introduced in the Solo distribution of Istio version 1.28 to disable the community multicluster mechanisms. Multicluster in community Istio uses remote secrets that contain kubeconfigs to watch resources on remote clusters. This system is incompatible with the decentralized, push-based model for peering in the Solo distribution of Istio. This variable causes istiod to ignore remote secrets so that it does not attempt to set up Kubernetes clients to connect to them.

  • For fresh multicluster mesh installations with the Solo distribution of Istio, use this environment variable in your istiod settings. This setting serves as a recommended safety measure to prevent any use of remote secrets.
  • If you want to initiate a multicluster migration from community Istio, contact a Solo account representative. An account representative can help you set up two revisions of Istio that each select a different set of namespaces, and set the DISABLE_LEGACY_MULTICLUSTER variable on the revision that uses the Solo distribution of Istio for multicluster peering.