Helm
Use Helm to link ambient service meshes across multiple clusters.
Overview
In this guide, you deploy an ambient mesh to each workload cluster, create an east-west gateway in each cluster, and link the istiod control planes across cluster networks by using peering gateways. In the next guide, you can deploy the Bookinfo sample app to the ambient mesh in each cluster, and make select services available across the multicluster mesh. Incoming requests can then be routed from an ingress gateway, such as Solo Enterprise for kgateway, to services in your mesh across all clusters.
The following diagram demonstrates an ambient mesh setup across multiple clusters. For more information about the components that are installed in these steps, see the ambient components overview.
Considerations
Before you set up a multicluster ambient mesh, review the following considerations and requirements.
License requirements
Version requirements
Review the following known Istio version requirements and restrictions.
- If you use Istio versions 1.27.7, 1.28.4, 1.29.0 or later, and you install the Solo Enterprise for Istio management plane into a namespace other than
gloo-mesh, you must allow that namespace by listing it in theDEBUG_ENDPOINT_AUTH_ALLOWED_NAMESPACESenvironment variable of your istiod installation. For more information, see the release notes. - Patch versions 1.26.0 and 1.26.1 of the Solo distribution of Istio lack support for FIPS-tagged images and ztunnel outlier detection. When upgrading or installing 1.26, be sure to use patch version
1.26.1-patch0and later only. - In the Solo distribution of Istio 1.25 and later, you can access enterprise-level features by passing your Solo license in the
license.valueorlicense.secretReffield of the Solo distribution of the istiod Helm chart. The Solo istiod Helm chart is strongly recommended due to the included safeguards, default settings, and upgrade handling to ensure a reliable and secure Istio deployment. Though it is not recommended, you can pass your license key in the open source istiod Helm chart by using the--set pilot.env.SOLO_LICENSE_KEYfield. - Multicluster setups require the Solo distribution of Istio version 1.24.3 or later (
1.24.3-solo), including the Solo distribution ofistioctl. - Due to a lack of support for the Istio CNI and iptables for the Istio proxy, you cannot run Istio (and therefore Solo Enterprise for Istio) on AWS Fargate. For more information, see the Amazon EKS issue.
Platform requirements
The steps in the following sections have options for deploying an ambient mesh to either Kubernetes or OpenShift clusters.
If you use OpenShift clusters, complete the following steps before you begin:
- Set
routingViaHost: truein the Cluster Network Operator to enable OVN-Kubernetes local gateway mode. For more information, see the Platform-specific prerequisites in the ambient mesh docs.
The commands for OpenShift in the following steps contain these required settings:
- Your Helm settings must include
global.platform=openshiftfor Istio 1.24 and later. If you instead install Istio 1.23 or earlier, you must useprofile=openshiftinstead of theglobal.platformsetting. - Install the
istio-cniandztunnelHelm releases in thekube-systemnamespace, instead of theistio-systemnamespace.
Revision and canary upgrade limitations
The upgrade guides in this documentation show you how to perform in-place upgrades for your Istio components, which is the recommended upgrade strategy.
Cross-cluster traffic addresses
In each cluster, you create an east-west gateway, which is implemented as a ztunnel that facilitates traffic between services across clusters in your multicluster mesh. In the Solo distribution of Istio 1.28 and later, you can use either LoadBalancer or NodePort addresses to resolve cross-cluster traffic requests through this gateway. Note that the NodePort method is considered alpha in the Solo distribution of Istio version 1.28.
LoadBalancer: In the standard LoadBalancer peering method, cross-cluster traffic through the east-west gateway resolves to its LoadBalancer address.
NodePort (alpha): If you prefer to use direct pod-to-pod traffic across clusters, you can annotate the east-west and peering gateways so that cross-cluster traffic resolves to NodePort addresses. This method allows you to avoid LoadBalancer services to reduce cross-cluster traffic costs. Review the following considerations:
- Note that the gateways must still be created with stable IP addresses, which are required for xDS communication with the istiod control plane in each cluster. NodePort peering is used for data-plane communication, in that requests to services resolve to the NodePort instead of the LoadBalancer address. Also, the east-west gateway must have the
topology.istio.io/clusterlabel. - If a node in a target cluster becomes inaccessible, such as during a restart or replacement, a delay can occur in the connection from the client cluster that must become aware of the new east-west gateway NodePort. In this case, you might see a connection error when trying to send cross-cluster traffic to an east-west gateway that is no longer accepting connections.
- Only nodes where an east-west gateway pod is provisioned are considered targets for traffic.
- NodePort peering uses only InternalIP node addresses. Ensure that your environment is configured so that nodes are reachable via their InternalIP address. If you need to use another address type, such as ExternalIP, contact Solo for engineering design and implementation support.
- Like LoadBalancer gateways, NodePort gateways support traffic from Envoy-based ingress gateways, waypoints, and sidecars.
- This feature is in an alpha state. Alpha features are likely to change, are not fully tested, and are not supported for production. For more information, see Solo feature maturity.
- This feature is only supported for default network setups, and is not applicable in flat network setups.
The steps in the following guide include options for either the LoadBalancer or NodePort method. A status condition on each east-west and remote peer gateway indicates which dataplane service type is in use.
Migrating from multicluster community Istio
If you previously used the multicluster feature in community Istio, and want to now migrate to multicluster peering in the Solo distribution of Istio, the DISABLE_LEGACY_MULTICLUSTER environment variable is introduced in the Solo distribution of Istio version 1.28 to disable the community multicluster mechanisms. Multicluster in community Istio uses remote secrets that contain kubeconfigs to watch resources on remote clusters. This system is incompatible with the decentralized, push-based model for peering in the Solo distribution of Istio. This variable causes istiod to ignore remote secrets so that it does not attempt to set up Kubernetes clients to connect to them.
- For fresh multicluster mesh installations with the Solo distribution of Istio, use this environment variable in your istiod settings. This setting serves as a recommended safety measure to prevent any use of remote secrets.
- If you want to initiate a multicluster migration from community Istio, contact a Solo account representative. An account representative can help you set up two revisions of Istio that each select a different set of namespaces, and set the
DISABLE_LEGACY_MULTICLUSTERvariable on the revision that uses the Solo distribution of Istio for multicluster peering.
Multicluster setup method
To get started, choose one of following methods for creating a multicluster mesh.
- Option 1: Install and link new ambient meshes to form a multicluster mesh.
- Option 2: Upgrade and link existing ambient meshes that you already installed in individual clusters.
- Option 3: Automatically link clusters by using the Gloo management plane. Note that this method is a beta feature.
Option 1: Install and link new ambient meshes
In each cluster, use Helm to create the ambient mesh components. Then, create an east-west gateway so that traffic requests can be routed cross-cluster, and link clusters to enable cross-cluster service discovery.
Set up tools
Set your Enterprise level license for Solo Enterprise for Istio as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing. Note that you might have previously saved this key in another variable, such as
${SOLO_LICENSE_KEY}or${GLOO_MESH_LICENSE_KEY}.export SOLO_ISTIO_LICENSE_KEY=<enterprise_license_key>Choose the version of Istio that you want to install or upgrade to by reviewing the supported versions.
Save the Solo distribution of Istio version.
export ISTIO_VERSION=1.28.5 export ISTIO_IMAGE=${ISTIO_VERSION}-soloSave the repo key for the minor version of the Solo distribution of Istio that you want to install. This is the 12-character hash at the end of the repo URL
us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Istio images built by Solo.io support article.# 12-character hash at the end of the repo URL export REPO_KEY=<repo_key> export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY} export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}Get the Solo distribution of Istio binary and install
istioctl, which you use for multicluster linking and gateway commands. This script automatically detects your OS and architecture, downloads the appropriate Solo distribution of Istio binary, and verifies the installation.bash <(curl -sSfL https://raw.githubusercontent.com/solo-io/doc-examples/main/istio/install-istioctl.sh) export PATH=${HOME}/.istioctl/bin:${PATH}Save the names and kubeconfig contexts of each cluster. This guide uses two clusters as an example. To add more clusters to the multicluster setup, include them in the arrays.
export cluster1=<cluster1_name> export context1=<cluster1_context> export cluster2=<cluster2_name> export context2=<cluster2_context>
Create a shared root of trust
Each cluster in the multicluster setup must have a shared root of trust. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.
By default, the Istio CA generates a self-signed root certificate and key, and uses them to sign the workload certificates. For more information, see the Plug in CA Certificates guide in the community Istio documentation.
For demo installations, you can run the following function to quickly generate and plug in the certificates and key for the Istio CA:
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}
mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca
function create_cacerts_secret() {
context=${1:?context}
cluster=${2:?cluster}
make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
kubectl --context=${context} create ns istio-system || true
kubectl --context=${context} create secret generic cacerts -n istio-system \
--from-file=${cluster}/ca-cert.pem \
--from-file=${cluster}/ca-key.pem \
--from-file=${cluster}/root-cert.pem \
--from-file=${cluster}/cert-chain.pem
}
create_cacerts_secret ${context1} ${cluster1}
create_cacerts_secret ${context2} ${cluster2}
cd ../..To enhance the security of your setup even further and have full control over the Istio CA lifecycle, you can generate and store the root and intermediate CA certificates and keys with your own PKI provider. You can then use tools such as cert-manager to send certificate signing requests on behalf of istiod to your PKI provider. Cert-manager stores the signed intermediate certificates and keys in the cacerts Kubernetes secret so that istiod can use these credentials to issue leaf certificates for the workloads in the service mesh. You can set up cert-manager to also check the certificates and renew them before they expire.
AWS Private CA issuer and cert-manager: For an architectural overview of this certificate setup, see Bring your own Istio CAs with AWS. For steps on how to deploy this certificate setup, check out this Solo.io blog post. Be sure to repeat the steps so that a cacerts secret exists in each cluster.
Deploy ambient components
Apply the CRDs for the Kubernetes Gateway API to your cluster, which are required to create components such as waypoint proxies for L7 traffic policies, gateways with the
Gatewayresource, and more.kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml --context ${context1} kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml --context ${context2}Install the
basechart, which contains the CRDs and cluster roles required to set up Istio, in both clusters.function install_base() { context=${1:?context} helm upgrade --install istio-base oci://${HELM_REPO}/base \ --namespace istio-system \ --create-namespace \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF defaultRevision: "" profile: ambient EOF } install_base ${context1} install_base ${context2}function install_base() { context=${1:?context} helm upgrade --install istio-base oci://${HELM_REPO}/base \ --namespace istio-system \ --create-namespace \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF defaultRevision: "" profile: ambient global: platform: openshift EOF } install_base ${context1} install_base ${context2}You can optionally verify that the CRDs are successfully installed in both clusters.
kubectl get crds -l app.kubernetes.io/instance=istio-base --kube-context ${context1} kubectl get crds -l app.kubernetes.io/instance=istio-base --kube-context ${context2}Example output:
NAME CREATED AT authorizationpolicies.security.istio.io 2025-12-16T22:56:00Z destinationrules.networking.istio.io 2025-12-16T22:56:00Z envoyfilters.networking.istio.io 2025-12-16T22:56:00Z gateways.networking.istio.io 2025-12-16T22:56:00Z peerauthentications.security.istio.io 2025-12-16T22:56:00Z proxyconfigs.networking.istio.io 2025-12-16T22:56:00Z requestauthentications.security.istio.io 2025-12-16T22:56:00Z segments.admin.solo.io 2025-12-16T22:56:00Z serviceentries.networking.istio.io 2025-12-16T22:56:00Z sidecars.networking.istio.io 2025-12-16T22:56:00Z telemetries.telemetry.istio.io 2025-12-16T22:56:00Z virtualservices.networking.istio.io 2025-12-16T22:56:00Z wasmplugins.extensions.istio.io 2025-12-16T22:56:00Z workloadentries.networking.istio.io 2025-12-16T22:56:00Z workloadgroups.networking.istio.io 2025-12-16T22:56:00ZCreate the
istiodcontrol plane in both clusters.function install_istiod() { context=${1:?context} cluster=${2:?cluster} helm upgrade --install istiod oci://${HELM_REPO}/istiod \ --namespace istio-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF env: # Assigns IP addresses to multicluster services PILOT_ENABLE_IP_AUTOALLOCATE: "true" # Required when meshConfig.trustDomain is set PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true" # Disable community Istio multicluster mechanisms DISABLE_LEGACY_MULTICLUSTER: "true" global: hub: ${REPO} multiCluster: clusterName: ${cluster} network: ${cluster} proxy: clusterDomain: cluster.local tag: ${ISTIO_IMAGE} meshConfig: accessLogFile: /dev/stdout defaultConfig: proxyMetadata: ISTIO_META_DNS_CAPTURE: "true" # Assign each cluster a unique trust domain to apply policies to specific clusters trustDomain: "${cluster}.local" pilot: cni: namespace: istio-system enabled: true # Required to enable multicluster support platforms: peering: enabled: true profile: ambient license: value: ${SOLO_ISTIO_LICENSE_KEY} # Uncomment if you prefer to specify your license secret # instead of an inline value. # secretRef: # name: # namespace: EOF } install_istiod ${context1} ${cluster1} install_istiod ${context2} ${cluster2}function install_istiod() { context=${1:?context} cluster=${2:?cluster} helm upgrade --install istiod oci://${HELM_REPO}/istiod \ --namespace istio-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF env: # Assigns IP addresses to multicluster services PILOT_ENABLE_IP_AUTOALLOCATE: "true" # Required when meshConfig.trustDomain is set PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true" # Disable community Istio multicluster mechanisms DISABLE_LEGACY_MULTICLUSTER: "true" global: hub: ${REPO} multiCluster: clusterName: ${cluster} network: ${cluster} platform: openshift proxy: clusterDomain: cluster.local tag: ${ISTIO_IMAGE} meshConfig: accessLogFile: /dev/stdout defaultConfig: proxyMetadata: ISTIO_META_DNS_CAPTURE: "true" # Assign each cluster a unique trust domain to apply policies to specific clusters trustDomain: "${cluster}.local" pilot: cni: namespace: kube-system enabled: true # Required to enable multicluster support platforms: peering: enabled: true profile: ambient license: value: ${SOLO_ISTIO_LICENSE_KEY} # Uncomment if you prefer to specify your license secret # instead of an inline value. # secretRef: # name: # namespace: EOF } install_istiod ${context1} ${cluster1} install_istiod ${context2} ${cluster2}Install the Istio CNI node agent daemonset in both clusters.
function install_cni() { context=${1:?context} helm upgrade --install istio-cni oci://${HELM_REPO}/cni \ --namespace istio-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF # Assigns IP addresses to multicluster services ambient: dnsCapture: true excludeNamespaces: - istio-system - kube-system global: hub: ${REPO} tag: ${ISTIO_IMAGE} profile: ambient EOF } install_cni ${context1} install_cni ${context2}function install_cni() { context=${1:?context} helm upgrade --install istio-cni oci://${HELM_REPO}/cni \ --namespace kube-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF # Assigns IP addresses to multicluster services ambient: dnsCapture: true excludeNamespaces: - istio-system - kube-system global: hub: ${REPO} platform: openshift tag: ${ISTIO_IMAGE} profile: ambient EOF } install_cni ${context1} install_cni ${context2}Install the ztunnel daemonset in both clusters.
function install_ztunnel() { context=${1:?context} cluster=${2:?cluster} helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \ --namespace istio-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF configValidation: true enabled: true env: L7_ENABLED: "true" # Required when a unique trust domain is set for each cluster SKIP_VALIDATE_TRUST_DOMAIN: "true" hub: ${REPO} istioNamespace: istio-system multiCluster: clusterName: ${cluster} namespace: istio-system network: ${cluster} profile: ambient proxy: clusterDomain: cluster.local tag: ${ISTIO_IMAGE} terminationGracePeriodSeconds: 29 variant: distroless EOF } install_ztunnel ${context1} ${cluster1} install_ztunnel ${context2} ${cluster2}function install_ztunnel() { context=${1:?context} cluster=${2:?cluster} helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \ --namespace kube-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF configValidation: true enabled: true env: L7_ENABLED: "true" # Required when a unique trust domain is set for each cluster SKIP_VALIDATE_TRUST_DOMAIN: "true" global: platform: openshift hub: ${REPO} istioNamespace: istio-system multiCluster: clusterName: ${cluster} namespace: kube-system network: ${cluster} profile: ambient proxy: clusterDomain: cluster.local tag: ${ISTIO_IMAGE} terminationGracePeriodSeconds: 29 variant: distroless EOF } install_ztunnel ${context1} ${cluster1} install_ztunnel ${context2} ${cluster2}Verify that the components of the Istio ambient control and data plane are successfully installed in both clusters. Because the Istio CNI and ztunnel are deployed as daemon sets, the number of CNI and ztunnel pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.
kubectl get pods -A --context ${context1} | grep -E 'istio|ztunnel' kubectl get pods -A --context ${context2} | grep -E 'istio|ztunnel'Example output:
istiod-85c4dfd97f-mncj5 1/1 Running 0 40s istio-cni-node-pr5rl 1/1 Running 0 9s istio-cni-node-pvmx2 1/1 Running 0 9s istio-cni-node-6q26l 1/1 Running 0 9s ztunnel-tvtzn 1/1 Running 0 7s ztunnel-vtpjm 1/1 Running 0 4s ztunnel-hllxg 1/1 Running 0 4sOptional: Check the istiod logs in both clusters to verify that the certificate you generated earlier is picked up by istiod.
kubectl logs deploy/istiod -n istio-system --context ${context1} | grep x509 kubectl logs deploy/istiod -n istio-system --context ${context2} | grep x509Example output:
2025-12-16T22:59:06.783901Z info x509 cert - Issuer: "CN=Intermediate CA,O=Istio,L=cluster-1", Subject: "", SN: def320623729b8370172413749143836, NotBefore: "2025-12-16T22:57:06Z", NotAfter: "2035-12-14T22:59:06Z" 2025-12-16T22:59:06.783937Z info x509 cert - Issuer: "CN=Root CA,O=Istio", Subject: "CN=Intermediate CA,O=Istio,L=cluster-1", SN: 452d45254328667ccf8434c64c79fe789612bb5a, NotBefore: "2025-12-16T22:54:30Z", NotAfter: "2035-12-14T22:54:30Z" 2025-12-16T22:59:06.783966Z info x509 cert - Issuer: "CN=Root CA,O=Istio", Subject: "CN=Root CA,O=Istio", SN: 38dad68e56ab8c506ae07454801f65b134bd9580, NotBefore: "2025-12-16T22:54:30Z", NotAfter: "2035-12-14T22:54:30Z"Label the
istio-systemnamespace with the clusters’ network names, which you previously set to each cluster name in theglobal.networkfield of theistiodinstallations. The ambient control plane uses this label internally to group pods that exist in the same L3 network.kubectl label namespace istio-system --context ${context1} topology.istio.io/network=${cluster1} kubectl label namespace istio-system --context ${context2} topology.istio.io/network=${cluster2}Create an east-west gateway in the
istio-eastwestnamespace. In each cluster, the east-west gateway is implemented as a ztunnel that facilitates traffic between services across clusters in your multicluster mesh.You can use either LoadBalancer or NodePort addresses for cross-cluster traffic.
Create the east-west gateway in both clusters. Cross-cluster traffic though this gateway resolves to the LoadBalancer address. For customization options, see the gateway guide in the Istio docs.
- Solo distribution of
istioctl: For more information about this command, see the CLI reference.function create_ew_gateway_lb() { context=${1:?context} cluster=${2:?cluster} kubectl create namespace istio-eastwest --context ${context} istioctl multicluster expose --namespace istio-eastwest --context ${context} --generate > ew-gateway-${cluster}.yaml kubectl apply -f ew-gateway-${cluster}.yaml --context ${context} } create_ew_gateway_lb ${context1} ${cluster1} create_ew_gateway_lb ${context2} ${cluster2}
Create the east-west gateway in both clusters. Note that the gateway must still be created with a stable internal IP address, which is required for xDS communication with the istiod control plane in each cluster. NodePort peering is used for data plane communication, in that requests to services resolve to the NodePort instead of the gateway’s stable IP address. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.
- Solo distribution of
istioctl: For more information about this command, see the CLI reference.function create_ew_gateway_nodeport() { context=${1:?context} cluster=${2:?cluster} kubectl create namespace istio-eastwest --context ${context} istioctl multicluster expose --namespace istio-eastwest --context ${context} --generate > ew-gateway-${cluster}.yaml kubectl apply -f ew-gateway-${cluster}.yaml --context ${context} kubectl annotate gateway istio-eastwest -n istio-eastwest peering.solo.io/data-plane-service-type=NodePort --context ${context} } create_ew_gateway_nodeport ${context1} ${cluster1} create_ew_gateway_nodeport ${context2} ${cluster2}
- Solo distribution of
Verify that the east-west gateway is successfully deployed in both clusters.
kubectl get svc -n istio-eastwest --context ${context1} kubectl get svc -n istio-eastwest --context ${context2}Example output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE istio-eastwest LoadBalancer 172.20.205.104 <external_address> 15021:31655/TCP,15008:32699/TCP,15012:32166/TCP 55s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE istio-eastwest LoadBalancer 172.20.21.117 <external_address> 15021:30324/TCP,15008:31875/TCP,15012:31050/TCP 77s
Link clusters
Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters.
Optional: Before you link clusters, you can check the individual readiness of each cluster for linking by running the
istioctl multicluster check --precheckcommand. For more information about this command, see the CLI reference. If any checks fail, run the command with--verbose, and see Validate your multicluster setup.istioctl multicluster check --precheck --contexts="$context1,$context2"Before continuing to the next step, make sure that the following checks pass as expected:✅ Relevant environment variables on istiod are supported.✅ The license in use by istiod supports multicluster.✅ All istiod, ztunnel, and east-west gateway pods are healthy.✅ The east-west gateway is programmed.
Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters. Note that you can either link the clusters bi-directionally or asymmetrically. In a standard bi-directional setup, services in any of the linked clusters can send requests to and receive requests from the services in any of the other linked clusters. In an asymmetrical setup, you allow one cluster to send requests to another cluster, but the other cluster cannot send requests back to the first cluster.
In the Solo distribution of Istio 1.29 or later, you can use the
peeringHelm chart to link your clusters.Get the addresses of the east-west gateway in each cluster. The following commands show examples for two clusters.
export CLUSTER1_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context1} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") echo "Cluster1 east-west gateway: $CLUSTER1_EW_ADDRESS" echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"Link the clusters by creating Helm releases in each cluster to represent the other clusters. In each cluster where you create Helm releases, a Gateway resource is created that uses the
istio-remoteGatewayClass. This class allows the gateway to connect to other clusters by using the addresses of the east-west gateways.- Be sure to update the region, and optionally zone, of each cluster.
- If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, change the
preferredDataplaneServiceTypetonodeport. This automatically configures the peering gateways for NodePort-based cross-cluster traffic. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses. - The following example depicts a bi-directional setup. In an asymmetrical setup, you create Helm releases to only represent the directionality you want to allow.
helm upgrade -i peering-remote oci://${HELM_REPO}/peering \ --version ${ISTIO_IMAGE} \ --namespace istio-eastwest \ --kube-context ${context1} \ -f - <<EOF remote: create: true items: # Remote peer configuration for cluster2 - name: istio-remote-peer-${cluster2} cluster: ${cluster2} network: ${cluster2} # Change to "Hostname" if using hostname addressType: IPAddress address: ${CLUSTER2_EW_ADDRESS} # Change to "nodeport" if using NodePorts preferredDataplaneServiceType: loadbalancer trustDomain: cluster.local region: "<cluster2_region>" # Uncomment as needed # zone: "<cluster2_zone>" EOF helm upgrade -i peering-remote oci://${HELM_REPO}/peering \ --version ${ISTIO_IMAGE} \ --namespace istio-eastwest \ --kube-context ${context2} \ -f - <<EOF remote: create: true items: # Remote peer configuration for cluster1 - name: istio-remote-peer-${cluster1} cluster: ${cluster1} network: ${cluster1} # Change to "Hostname" if using hostname addressType: IPAddress address: ${CLUSTER1_EW_ADDRESS} # Change to "nodeport" if using NodePorts preferredDataplaneServiceType: loadbalancer trustDomain: cluster.local region: "<cluster1_region>" # Uncomment as needed # zone: "<cluster1_zone>" EOF
Use the
istioctl multicluster linkcommand to quickly link clusters.Verify that the contexts for the clusters that you want to include in the multicluster mesh are listed in your kubeconfig file, which is required for the
istioctl multicluster linkcommand. If you do not have access to the kubeconfig files, use the declarative resources tab.kubectl config get-contexts- In the output, note the names of the cluster contexts, which you use in the next step to link the clusters.
- If you have multiple kubeconfig files, you can generate a merged kubeconfig file by running the following command.
KUBECONFIG=<kubeconfig_file1>.yaml:<file2>.yaml kubectl config view --flatten
Using the names of the cluster contexts, link the clusters so that they can communicate. To take a look at the Gateway resources that this command creates, you can include the
--generateflag in the command. For more information about this command, see the CLI reference.- Bi-directional: You can use the following
istioctlcommand to quickly link the clusters bi-directionally. In each cluster, Gateway resources are created that use theistio-remoteGatewayClass. This class allows the gateways to connect to other clusters by using the addresses of the east-west gateways.istioctl multicluster link --namespace istio-eastwest --contexts="$context1,$context2" ```<pre><code> If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, use the `--preferred-data-plane-service-type nodeport` flag when linking clusters. This automatically configures the peering gateways for NodePort-based cross-cluster traffic. For more information about NodePort peering considerations and requirements, see [Cross-cluster traffic addresses](#cross-cluster-traffic-addresses).
- Bi-directional: You can use the following
istioctl multicluster link --namespace istio-eastwest --contexts="$context1,$context2" --preferred-data-plane-service-type nodeportExample output for two clusters: ``` Gateway istio-eastwest/istio-remote-peer-cluster1 applied to cluster "cluster2" pointing to cluster "cluster1" (network "cluster1") Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "cluster1" pointing to cluster "cluster2" (network "cluster2") ```Asymmetrical: You can use the following
istioctlcommand to quickly link the clusters asymmetrically. The services in the cluster in the--fromflag can send requests to services in the cluster in the--toflag, but sending requests in the reverse direction is not permitted.For example, this command allows services in
cluster1’s mesh to send requests to services incluster2’s mesh throughcluster2’s east-west gateway. However, the reverse is not permitted: services incluster2’s mesh cannot send requests throughcluster1’s east-west gateway to services incluster1.istioctl multicluster link --namespace istio-eastwest --from ${context1} --to ${context2} ```<pre><code> If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, use the `--preferred-data-plane-service-type nodeport` flag when linking clusters. This automatically configures the peering gateways for NodePort-based cross-cluster traffic. For more information about NodePort peering considerations and requirements, see [Cross-cluster traffic addresses](#cross-cluster-traffic-addresses).
istioctl multicluster link --namespace istio-eastwest --from ${context1} --to ${context2} --preferred-data-plane-service-type nodeportExample output: ``` Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "cluster1" pointing to cluster "cluster2" (network "cluster2") ```Link the clusters by declaratively creating
istio-remotepeer gateways.Bi-directional: Use the following Gateway resources to create an
istio-remotepeer gateway in each cluster. Theistio-remoteGatewayClass allows the gateways to connect to other clusters by using the addresses of the east-west gateways.Get the addresses of the east-west gateway in each cluster. The following commands show examples for two clusters.
export CLUSTER1_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context1} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") echo "Cluster1 east-west gateway: $CLUSTER1_EW_ADDRESS" echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"Using the east-west gateway addresses, create a Gateway resource in each cluster to represent the other cluster.
- In the
labelssection, be sure to update the locality labels according to the region, and optionally zone, of each cluster. For more information about locality support, see the release notes. - If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, uncomment the
peering.solo.io/preferred-data-plane-service-type: NodePortannotation from each Gateway resource. Additionally, you can comment out the HBONE listener in each gateway, because traffic is routed through the NodePort directly. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.
kubectl apply --context ${context1} -f- <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: annotations: gateway.istio.io/service-account: istio-eastwest gateway.istio.io/trust-domain: ${cluster2} # Uncomment for NodePort-based cross-cluster traffic # peering.solo.io/preferred-data-plane-service-type: NodePort labels: topology.istio.io/network: ${cluster2} topology.kubernetes.io/region: "<cluster2_region>" # Uncomment as needed # topology.kubernetes.io/zone: "<cluster2_zone>" name: istio-remote-peer-${cluster2} namespace: istio-eastwest spec: addresses: # Change to 'type: Hostname' as needed, such as for AWS hostnames - type: IPAddress value: $CLUSTER2_EW_ADDRESS gatewayClassName: istio-remote listeners: # Comment HBONE out for NodePort-based cross-cluster traffic - name: cross-network port: 15008 protocol: HBONE tls: mode: Passthrough - name: xds-tls port: 15012 protocol: TLS tls: mode: Passthrough EOF kubectl apply --context ${context2} -f- <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: annotations: gateway.istio.io/service-account: istio-eastwest gateway.istio.io/trust-domain: ${cluster1} # Uncomment for NodePort-based cross-cluster traffic # peering.solo.io/preferred-data-plane-service-type: NodePort labels: topology.istio.io/network: ${cluster1} topology.kubernetes.io/region: "<cluster1_region>" # Uncomment as needed # topology.kubernetes.io/zone: "<cluster1_zone>" name: istio-remote-peer-${cluster1} namespace: istio-eastwest spec: addresses: # Change to 'type: Hostname' as needed, such as for AWS hostnames - type: IPAddress value: $CLUSTER1_EW_ADDRESS gatewayClassName: istio-remote listeners: # Comment HBONE out for NodePort-based cross-cluster traffic - name: cross-network port: 15008 protocol: HBONE tls: mode: Passthrough - name: xds-tls port: 15012 protocol: TLS tls: mode: Passthrough EOF- In the
Asymmetrical: Use the following Gateway resources to create an
istio-remotepeer gateway in only some clusters. Theistio-remoteGatewayClass allows the gateway in one cluster to connect to another cluster by using the address of the east-west gateway, but sending requests in the reverse direction is not permitted.Get the address of the east-west gateway in the cluster that you want to send traffic to. The following command shows an example for
cluster2.export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"Using the east-west gateway address, create a Gateway resource in the cluster that you want to send requests from. For example, this Gateway resource allows services in
cluster1’s mesh to send requests to services incluster2’s mesh throughcluster2’s east-west gateway. However, the reverse is not permitted: services incluster2’s mesh cannot send requests throughcluster1’s east-west gateway to services incluster1.- In the
labelssection, be sure to update the locality labels according to the region, and optionally zone, of each cluster. For more information about locality support, see the release notes. - If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, uncomment the
peering.solo.io/preferred-data-plane-service-type: NodePortannotation from each Gateway resource. Additionally, you can comment out the HBONE listener in each gateway, because traffic is routed through the NodePort directly. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.
kubectl apply --context ${context1} -f- <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: annotations: gateway.istio.io/service-account: istio-eastwest gateway.istio.io/trust-domain: ${cluster2} # Uncomment for NodePort-based cross-cluster traffic # peering.solo.io/preferred-data-plane-service-type: NodePort labels: topology.istio.io/network: ${cluster2} topology.kubernetes.io/region: "<cluster2_region>" # Uncomment as needed # topology.kubernetes.io/zone: "<cluster2_zone>" name: istio-remote-peer-${cluster2} namespace: istio-eastwest spec: addresses: # Change to 'type: Hostname' as needed, such as for AWS hostnames - type: IPAddress value: $CLUSTER2_EW_ADDRESS gatewayClassName: istio-remote listeners: # Comment HBONE out for NodePort-based cross-cluster traffic - name: cross-network port: 15008 protocol: HBONE tls: mode: Passthrough - name: xds-tls port: 15012 protocol: TLS tls: mode: Passthrough EOF- In the
Verify that peer linking was successful by running the
istioctl multicluster checkcommand. If any checks fail, run the command with--verbose, and see Validate your multicluster setup.istioctl multicluster check --contexts="$context1,$context2"In this example output, the remote peer gateways are successfully connected, and all other checks passed successfully. No global services exist because no app services are exposed across clusters in the multicluster mesh yet.
=== Cluster: cluster1 === ✅ Incompatible Environment Variable Check: all relevant environment variables are valid ✅ License Check: license is valid for multicluster ✅ CNI DNS Capture Check: AMBIENT_DNS_CAPTURE is enabled ✅ Pod Check (istiod): all pods healthy ✅ Pod Check (ztunnel): all pods healthy ✅ Pod Check (eastwest gateway istio-eastwest/istio-eastwest): all pods healthy ✅ Gateway Check: all eastwest gateways programmed ✅ istio-eastwest/istio-eastwest available at aab8471c7fcfa4a3c82f2d217b015d97-396238517.us-east-1.elb.amazonaws.com ✅ Peers Check: all clusters connected ✅ Connected to gloo-gateway-docs-mgt via ab46aa29a49914da789a6d5422aca279-541415195.us-east-2.elb.amazonaws.com ℹ️ Shared Services Check: no globally shared services found ====== === Cluster: cluster2 === ✅ Incompatible Environment Variable Check: all relevant environment variables are valid ✅ License Check: license is valid for multicluster ✅ CNI DNS Capture Check: AMBIENT_DNS_CAPTURE is enabled ✅ Pod Check (istiod): all pods healthy ✅ Pod Check (ztunnel): all pods healthy ✅ Pod Check (eastwest gateway istio-eastwest/istio-eastwest): all pods healthy ✅ Gateway Check: all eastwest gateways programmed ✅ istio-eastwest/istio-eastwest available at ab46aa29a49914da789a6d5422aca279-541415195.us-east-2.elb.amazonaws.com ✅ Peers Check: all clusters connected ✅ Connected to gloo-mesh-core-docs-mgt via aab8471c7fcfa4a3c82f2d217b015d97-396238517.us-east-1.elb.amazonaws.com ℹ️ Shared Services Check: no globally shared services found ====== ✅ Intermediate Certs Compatibility Check: all clusters have compatible intermediate certificates ✅ Network Configuration Check: all network configurations are valid ✅ Stale Workloads Check: skipped (flat network not detected)Optional: Verify that the istiod control plane for each peered cluster is included in each cluster’s proxy status list.
istioctl proxy-status --context ${context1} istioctl proxy-status --context ${context2}Example output for
cluster1, in which you can verify that the istiod control plane forcluster2is listed:NAME CLUSTER ISTIOD VERSION SUBSCRIBED TYPES istio-eastwest-67fd5679dc-fhsxs.istio-eastwest cluster1 istiod-7b7c9cc4c6-bdm9c 1.28.5-solo-fips 2 (WADS,WDS) istiod-6bc6765484-5bbhd.istio-system cluster2 istiod-7b7c9cc4c6-bdm9c 1.28.5-solo-fips 3 (FSDS,SGDS,WDS) ztunnel-5f8rb.kube-system cluster1 istiod-7b7c9cc4c6-bdm9c 1.28.5-solo-fips 2 (WADS,WDS) ztunnel-f96kh.kube-system cluster1 istiod-7b7c9cc4c6-bdm9c 1.28.5-solo-fips 2 (WADS,WDS) ztunnel-vtj4f.kube-system cluster1 istiod-7b7c9cc4c6-bdm9c 1.28.5-solo-fips 2 (WADS,WDS)
Next: Add apps to the ambient mesh. For multicluster setups, this includes making specific services available across your linked cluster setup.
Option 2: Upgrade and link existing ambient meshes
Upgrade your existing ambient meshes installed with Helm and link them to create a multicluster ambient mesh.
Set up tools
Set your Enterprise level license for Solo Enterprise for Istio as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing. Note that you might have previously saved this key in another variable, such as
${SOLO_LICENSE_KEY}or${GLOO_MESH_LICENSE_KEY}.export SOLO_ISTIO_LICENSE_KEY=<enterprise_license_key>Choose the version of Istio that you want to install or upgrade to by reviewing the supported versions.
Save the Solo distribution of Istio version.
export ISTIO_VERSION=1.28.5 export ISTIO_IMAGE=${ISTIO_VERSION}-soloSave the repo key for the minor version of the Solo distribution of Istio that you want to install. This is the 12-character hash at the end of the repo URL
us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Istio images built by Solo.io support article.# 12-character hash at the end of the repo URL export REPO_KEY=<repo_key> export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY} export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}Get the Solo distribution of Istio binary and install
istioctl, which you use for multicluster linking and gateway commands. This script automatically detects your OS and architecture, downloads the appropriate Solo distribution of Istio binary, and verifies the installation.bash <(curl -sSfL https://raw.githubusercontent.com/solo-io/doc-examples/main/istio/install-istioctl.sh) export PATH=${HOME}/.istioctl/bin:${PATH}Save the names and kubeconfig contexts of each cluster. This guide uses two clusters as an example. To add more clusters to the multicluster setup, include them in the arrays.
export cluster1=<cluster1_name> export context1=<cluster1_context> export cluster2=<cluster2_name> export context2=<cluster2_context>
Create a shared root of trust
Each cluster in the multicluster setup must have a shared root of trust. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.
By default, the Istio CA generates a self-signed root certificate and key, and uses them to sign the workload certificates. For more information, see the Plug in CA Certificates guide in the community Istio documentation.
For demo installations, you can run the following function to quickly generate and plug in the certificates and key for the Istio CA:
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}
mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca
function create_cacerts_secret() {
context=${1:?context}
cluster=${2:?cluster}
make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
kubectl --context=${context} create ns istio-system || true
kubectl --context=${context} create secret generic cacerts -n istio-system \
--from-file=${cluster}/ca-cert.pem \
--from-file=${cluster}/ca-key.pem \
--from-file=${cluster}/root-cert.pem \
--from-file=${cluster}/cert-chain.pem
}
create_cacerts_secret ${context1} ${cluster1}
create_cacerts_secret ${context2} ${cluster2}
cd ../..To enhance the security of your setup even further and have full control over the Istio CA lifecycle, you can generate and store the root and intermediate CA certificates and keys with your own PKI provider. You can then use tools such as cert-manager to send certificate signing requests on behalf of istiod to your PKI provider. Cert-manager stores the signed intermediate certificates and keys in the cacerts Kubernetes secret so that istiod can use these credentials to issue leaf certificates for the workloads in the service mesh. You can set up cert-manager to also check the certificates and renew them before they expire.
AWS Private CA issuer and cert-manager: For an architectural overview of this certificate setup, see Bring your own Istio CAs with AWS. For steps on how to deploy this certificate setup, check out this Solo.io blog post. Be sure to repeat the steps so that a cacerts secret exists in each cluster.
Upgrade settings
In each cluster, update the ambient mesh components for multicluster, and create an east-west gateway so that traffic requests can be routed cross-cluster.
Get the current values for the istiod Helm release in both clusters.
function get_istiod_values() { context=${1:?context} cluster=${2:?cluster} helm get values --kube-context ${context} istiod -n istio-system -o yaml > istiod-${cluster}.yaml } get_istiod_values ${context1} ${cluster1} get_istiod_values ${context2} ${cluster2}Update your Helm release with the following multicluster values in both clusters. If you must update the Istio minor version, include the
--set global.tag=${ISTIO_IMAGE}and--set global.hub=${REPO}flags too.If you prefer to specify your license secret instead of an inline value, you can include
--set license.secretRef.name=<name>and--set license.secretRef.namespace=<namespace>.function upgrade_istiod() { context=${1:?context} cluster=${2:?cluster} helm upgrade istiod oci://${HELM_REPO}/istiod \ --namespace istio-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f istiod-${cluster}.yaml \ --set env.PILOT_ENABLE_IP_AUTOALLOCATE="true" \ --set env.DISABLE_LEGACY_MULTICLUSTER="true" \ --set env.PILOT_SKIP_VALIDATE_TRUST_DOMAIN="true" \ --set global.multiCluster.clusterName=${cluster} \ --set global.network=${cluster} \ --set global.trustDomain="${cluster}.local" \ --set platforms.peering.enabled=true \ --set license.value=${SOLO_ISTIO_LICENSE_KEY} } upgrade_istiod ${context1} ${cluster1} upgrade_istiod ${context2} ${cluster2}If you prefer to specify your license secret instead of an inline value, you can include
--set license.secretRef.name=<name>and--set license.secretRef.namespace=<namespace>.function upgrade_istiod() { context=${1:?context} cluster=${2:?cluster} helm upgrade istiod oci://${HELM_REPO}/istiod \ --namespace istio-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f istiod-${cluster}.yaml \ --set env.PILOT_ENABLE_IP_AUTOALLOCATE="true" \ --set env.DISABLE_LEGACY_MULTICLUSTER="true" \ --set env.PILOT_SKIP_VALIDATE_TRUST_DOMAIN="true" \ --set global.multiCluster.clusterName=${cluster} \ --set global.network=${cluster} \ --set global.platform=openshift \ --set global.trustDomain="${cluster}.local" \ --set platforms.peering.enabled=true \ --set license.value=${SOLO_ISTIO_LICENSE_KEY} } upgrade_istiod ${context1} ${cluster1} upgrade_istiod ${context2} ${cluster2}Verify that the istiod pods are successfully restarted in both clusters. Note that it might take a few seconds for the pods to become available.
kubectl get pods --context ${context1} -n istio-system | grep istiod kubectl get pods --context ${context2} -n istio-system | grep istiodExample output:
istiod-b84c55cff-tllfr 1/1 Running 0 58sOptional: Check the istiod logs in both clusters to verify that the certificate you generated earlier is picked up by istiod.
kubectl logs deploy/istiod -n istio-system --context ${context1} | grep x509 kubectl logs deploy/istiod -n istio-system --context ${context2} | grep x509Example output:
2025-12-16T22:59:06.783901Z info x509 cert - Issuer: "CN=Intermediate CA,O=Istio,L=cluster-1", Subject: "", SN: def320623729b8370172413749143836, NotBefore: "2025-12-16T22:57:06Z", NotAfter: "2035-12-14T22:59:06Z" 2025-12-16T22:59:06.783937Z info x509 cert - Issuer: "CN=Root CA,O=Istio", Subject: "CN=Intermediate CA,O=Istio,L=cluster-1", SN: 452d45254328667ccf8434c64c79fe789612bb5a, NotBefore: "2025-12-16T22:54:30Z", NotAfter: "2035-12-14T22:54:30Z" 2025-12-16T22:59:06.783966Z info x509 cert - Issuer: "CN=Root CA,O=Istio", Subject: "CN=Root CA,O=Istio", SN: 38dad68e56ab8c506ae07454801f65b134bd9580, NotBefore: "2025-12-16T22:54:30Z", NotAfter: "2035-12-14T22:54:30Z"Get the current values for the ztunnel Helm release in both clusters.
function get_ztunnel_values() { context=${1:?context} cluster=${2:?cluster} helm get values ztunnel --kube-context ${context} -n istio-system -o yaml > ztunnel-${cluster}.yaml } get_ztunnel_values ${context1} ${cluster1} get_ztunnel_values ${context2} ${cluster2}function get_ztunnel_values() { context=${1:?context} cluster=${2:?cluster} helm get values ztunnel --kube-context ${context} -n kube-system -o yaml > ztunnel-${cluster}.yaml } get_ztunnel_values ${context1} ${cluster1} get_ztunnel_values ${context2} ${cluster2}Update your Helm release with the following multicluster values in both clusters. If you must update the Istio minor version, include the
--set tag=${ISTIO_IMAGE}and--set hub=${REPO}flags too.function upgrade_ztunnel() { context=${1:?context} cluster=${2:?cluster} helm upgrade ztunnel oci://${HELM_REPO}/ztunnel \ -n istio-system \ --version ${ISTIO_IMAGE} \ --kube-context ${context} \ -f ztunnel-${cluster}.yaml \ --set env.SKIP_VALIDATE_TRUST_DOMAIN="true" \ --set multiCluster.clusterName=${cluster} \ --set network=${cluster} } upgrade_ztunnel ${context1} ${cluster1} upgrade_ztunnel ${context2} ${cluster2}function upgrade_ztunnel() { context=${1:?context} cluster=${2:?cluster} helm upgrade ztunnel oci://${HELM_REPO}/ztunnel \ -n kube-system \ --version ${ISTIO_IMAGE} \ --kube-context ${context} \ -f ztunnel-${cluster}.yaml \ --set env.SKIP_VALIDATE_TRUST_DOMAIN="true" \ --set multiCluster.clusterName=${cluster} \ --set network=${cluster} } upgrade_ztunnel ${context1} ${cluster1} upgrade_ztunnel ${context2} ${cluster2}Verify that the ztunnel pods are successfully installed in both clusters. Because the ztunnel is deployed as a daemon set, the number of pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.
kubectl get pods -A --context ${context1} | grep ztunnel kubectl get pods -A --context ${context2} | grep ztunnelExample output for one cluster:
ztunnel-tvtzn 1/1 Running 0 7s ztunnel-vtpjm 1/1 Running 0 4s ztunnel-hllxg 1/1 Running 0 4sLabel the
istio-systemnamespace with the clusters’ network names, which you previously set to each cluster name in theglobal.networkfield of theistiodinstallations. The ambient control plane uses this label internally to group pods that exist in the same L3 network.kubectl label namespace istio-system --context ${context1} topology.istio.io/network=${cluster1} kubectl label namespace istio-system --context ${context2} topology.istio.io/network=${cluster2}Create an east-west gateway in the
istio-eastwestnamespace. In each cluster, the east-west gateway is implemented as a ztunnel that facilitates traffic between services across clusters in your multicluster mesh.You can use either LoadBalancer or NodePort addresses for cross-cluster traffic.
Create the east-west gateway in both clusters. Cross-cluster traffic though this gateway resolves to the LoadBalancer address. For customization options, see the gateway guide in the Istio docs.
- Solo distribution of
istioctl: For more information about this command, see the CLI reference.function create_ew_gateway_lb() { context=${1:?context} cluster=${2:?cluster} kubectl create namespace istio-eastwest --context ${context} istioctl multicluster expose --namespace istio-eastwest --context ${context} --generate > ew-gateway-${cluster}.yaml kubectl apply -f ew-gateway-${cluster}.yaml --context ${context} } create_ew_gateway_lb ${context1} ${cluster1} create_ew_gateway_lb ${context2} ${cluster2}
Create the east-west gateway in both clusters. Note that the gateway must still be created with a stable internal IP address, which is required for xDS communication with the istiod control plane in each cluster. NodePort peering is used for data plane communication, in that requests to services resolve to the NodePort instead of the gateway’s stable IP address. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.
- Solo distribution of
istioctl: For more information about this command, see the CLI reference.function create_ew_gateway_nodeport() { context=${1:?context} cluster=${2:?cluster} kubectl create namespace istio-eastwest --context ${context} istioctl multicluster expose --namespace istio-eastwest --context ${context} --generate > ew-gateway-${cluster}.yaml kubectl apply -f ew-gateway-${cluster}.yaml --context ${context} kubectl annotate gateway istio-eastwest -n istio-eastwest peering.solo.io/data-plane-service-type=NodePort --context ${context} } create_ew_gateway_nodeport ${context1} ${cluster1} create_ew_gateway_nodeport ${context2} ${cluster2}
- Solo distribution of
Verify that the east-west gateway is successfully deployed in both clusters.
kubectl get svc -n istio-eastwest --context ${context1} kubectl get svc -n istio-eastwest --context ${context2}Example output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE istio-eastwest LoadBalancer 172.20.205.104 <external_address> 15021:31655/TCP,15008:32699/TCP,15012:32166/TCP 55s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE istio-eastwest LoadBalancer 172.20.21.117 <external_address> 15021:30324/TCP,15008:31875/TCP,15012:31050/TCP 77s
Link clusters
Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters.
Optional: Before you link clusters, you can check the individual readiness of each cluster for linking by running the
istioctl multicluster check --precheckcommand. For more information about this command, see the CLI reference. If any checks fail, run the command with--verbose, and see Validate your multicluster setup.istioctl multicluster check --precheck --contexts="$context1,$context2"Before continuing to the next step, make sure that the following checks pass as expected:✅ Relevant environment variables on istiod are supported.✅ The license in use by istiod supports multicluster.✅ All istiod, ztunnel, and east-west gateway pods are healthy.✅ The east-west gateway is programmed.
Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters. Note that you can either link the clusters bi-directionally or asymmetrically. In a standard bi-directional setup, services in any of the linked clusters can send requests to and receive requests from the services in any of the other linked clusters. In an asymmetrical setup, you allow one cluster to send requests to another cluster, but the other cluster cannot send requests back to the first cluster.
In the Solo distribution of Istio 1.29 or later, you can use the
peeringHelm chart to link your clusters.Get the addresses of the east-west gateway in each cluster. The following commands show examples for two clusters.
export CLUSTER1_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context1} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") echo "Cluster1 east-west gateway: $CLUSTER1_EW_ADDRESS" echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"Link the clusters by creating Helm releases in each cluster to represent the other clusters. In each cluster where you create Helm releases, a Gateway resource is created that uses the
istio-remoteGatewayClass. This class allows the gateway to connect to other clusters by using the addresses of the east-west gateways.- Be sure to update the region, and optionally zone, of each cluster.
- If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, change the
preferredDataplaneServiceTypetonodeport. This automatically configures the peering gateways for NodePort-based cross-cluster traffic. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses. - The following example depicts a bi-directional setup. In an asymmetrical setup, you create Helm releases to only represent the directionality you want to allow.
helm upgrade -i peering-remote oci://${HELM_REPO}/peering \ --version ${ISTIO_IMAGE} \ --namespace istio-eastwest \ --kube-context ${context1} \ -f - <<EOF remote: create: true items: # Remote peer configuration for cluster2 - name: istio-remote-peer-${cluster2} cluster: ${cluster2} network: ${cluster2} # Change to "Hostname" if using hostname addressType: IPAddress address: ${CLUSTER2_EW_ADDRESS} # Change to "nodeport" if using NodePorts preferredDataplaneServiceType: loadbalancer trustDomain: cluster.local region: "<cluster2_region>" # Uncomment as needed # zone: "<cluster2_zone>" EOF helm upgrade -i peering-remote oci://${HELM_REPO}/peering \ --version ${ISTIO_IMAGE} \ --namespace istio-eastwest \ --kube-context ${context2} \ -f - <<EOF remote: create: true items: # Remote peer configuration for cluster1 - name: istio-remote-peer-${cluster1} cluster: ${cluster1} network: ${cluster1} # Change to "Hostname" if using hostname addressType: IPAddress address: ${CLUSTER1_EW_ADDRESS} # Change to "nodeport" if using NodePorts preferredDataplaneServiceType: loadbalancer trustDomain: cluster.local region: "<cluster1_region>" # Uncomment as needed # zone: "<cluster1_zone>" EOF
Use the
istioctl multicluster linkcommand to quickly link clusters.Verify that the contexts for the clusters that you want to include in the multicluster mesh are listed in your kubeconfig file, which is required for the
istioctl multicluster linkcommand. If you do not have access to the kubeconfig files, use the declarative resources tab.kubectl config get-contexts- In the output, note the names of the cluster contexts, which you use in the next step to link the clusters.
- If you have multiple kubeconfig files, you can generate a merged kubeconfig file by running the following command.
KUBECONFIG=<kubeconfig_file1>.yaml:<file2>.yaml kubectl config view --flatten
Using the names of the cluster contexts, link the clusters so that they can communicate. To take a look at the Gateway resources that this command creates, you can include the
--generateflag in the command. For more information about this command, see the CLI reference.- Bi-directional: You can use the following
istioctlcommand to quickly link the clusters bi-directionally. In each cluster, Gateway resources are created that use theistio-remoteGatewayClass. This class allows the gateways to connect to other clusters by using the addresses of the east-west gateways.istioctl multicluster link --namespace istio-eastwest --contexts="$context1,$context2" ```<pre><code> If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, use the `--preferred-data-plane-service-type nodeport` flag when linking clusters. This automatically configures the peering gateways for NodePort-based cross-cluster traffic. For more information about NodePort peering considerations and requirements, see [Cross-cluster traffic addresses](#cross-cluster-traffic-addresses).
- Bi-directional: You can use the following
istioctl multicluster link --namespace istio-eastwest --contexts="$context1,$context2" --preferred-data-plane-service-type nodeportExample output for two clusters: ``` Gateway istio-eastwest/istio-remote-peer-cluster1 applied to cluster "cluster2" pointing to cluster "cluster1" (network "cluster1") Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "cluster1" pointing to cluster "cluster2" (network "cluster2") ```Asymmetrical: You can use the following
istioctlcommand to quickly link the clusters asymmetrically. The services in the cluster in the--fromflag can send requests to services in the cluster in the--toflag, but sending requests in the reverse direction is not permitted.For example, this command allows services in
cluster1’s mesh to send requests to services incluster2’s mesh throughcluster2’s east-west gateway. However, the reverse is not permitted: services incluster2’s mesh cannot send requests throughcluster1’s east-west gateway to services incluster1.istioctl multicluster link --namespace istio-eastwest --from ${context1} --to ${context2} ```<pre><code> If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, use the `--preferred-data-plane-service-type nodeport` flag when linking clusters. This automatically configures the peering gateways for NodePort-based cross-cluster traffic. For more information about NodePort peering considerations and requirements, see [Cross-cluster traffic addresses](#cross-cluster-traffic-addresses).
istioctl multicluster link --namespace istio-eastwest --from ${context1} --to ${context2} --preferred-data-plane-service-type nodeportExample output: ``` Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "cluster1" pointing to cluster "cluster2" (network "cluster2") ```Link the clusters by declaratively creating
istio-remotepeer gateways.Bi-directional: Use the following Gateway resources to create an
istio-remotepeer gateway in each cluster. Theistio-remoteGatewayClass allows the gateways to connect to other clusters by using the addresses of the east-west gateways.Get the addresses of the east-west gateway in each cluster. The following commands show examples for two clusters.
export CLUSTER1_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context1} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") echo "Cluster1 east-west gateway: $CLUSTER1_EW_ADDRESS" echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"Using the east-west gateway addresses, create a Gateway resource in each cluster to represent the other cluster.
- In the
labelssection, be sure to update the locality labels according to the region, and optionally zone, of each cluster. For more information about locality support, see the release notes. - If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, uncomment the
peering.solo.io/preferred-data-plane-service-type: NodePortannotation from each Gateway resource. Additionally, you can comment out the HBONE listener in each gateway, because traffic is routed through the NodePort directly. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.
kubectl apply --context ${context1} -f- <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: annotations: gateway.istio.io/service-account: istio-eastwest gateway.istio.io/trust-domain: ${cluster2} # Uncomment for NodePort-based cross-cluster traffic # peering.solo.io/preferred-data-plane-service-type: NodePort labels: topology.istio.io/network: ${cluster2} topology.kubernetes.io/region: "<cluster2_region>" # Uncomment as needed # topology.kubernetes.io/zone: "<cluster2_zone>" name: istio-remote-peer-${cluster2} namespace: istio-eastwest spec: addresses: # Change to 'type: Hostname' as needed, such as for AWS hostnames - type: IPAddress value: $CLUSTER2_EW_ADDRESS gatewayClassName: istio-remote listeners: # Comment HBONE out for NodePort-based cross-cluster traffic - name: cross-network port: 15008 protocol: HBONE tls: mode: Passthrough - name: xds-tls port: 15012 protocol: TLS tls: mode: Passthrough EOF kubectl apply --context ${context2} -f- <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: annotations: gateway.istio.io/service-account: istio-eastwest gateway.istio.io/trust-domain: ${cluster1} # Uncomment for NodePort-based cross-cluster traffic # peering.solo.io/preferred-data-plane-service-type: NodePort labels: topology.istio.io/network: ${cluster1} topology.kubernetes.io/region: "<cluster1_region>" # Uncomment as needed # topology.kubernetes.io/zone: "<cluster1_zone>" name: istio-remote-peer-${cluster1} namespace: istio-eastwest spec: addresses: # Change to 'type: Hostname' as needed, such as for AWS hostnames - type: IPAddress value: $CLUSTER1_EW_ADDRESS gatewayClassName: istio-remote listeners: # Comment HBONE out for NodePort-based cross-cluster traffic - name: cross-network port: 15008 protocol: HBONE tls: mode: Passthrough - name: xds-tls port: 15012 protocol: TLS tls: mode: Passthrough EOF- In the
Asymmetrical: Use the following Gateway resources to create an
istio-remotepeer gateway in only some clusters. Theistio-remoteGatewayClass allows the gateway in one cluster to connect to another cluster by using the address of the east-west gateway, but sending requests in the reverse direction is not permitted.Get the address of the east-west gateway in the cluster that you want to send traffic to. The following command shows an example for
cluster2.export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"Using the east-west gateway address, create a Gateway resource in the cluster that you want to send requests from. For example, this Gateway resource allows services in
cluster1’s mesh to send requests to services incluster2’s mesh throughcluster2’s east-west gateway. However, the reverse is not permitted: services incluster2’s mesh cannot send requests throughcluster1’s east-west gateway to services incluster1.- In the
labelssection, be sure to update the locality labels according to the region, and optionally zone, of each cluster. For more information about locality support, see the release notes. - If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, uncomment the
peering.solo.io/preferred-data-plane-service-type: NodePortannotation from each Gateway resource. Additionally, you can comment out the HBONE listener in each gateway, because traffic is routed through the NodePort directly. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.
kubectl apply --context ${context1} -f- <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: annotations: gateway.istio.io/service-account: istio-eastwest gateway.istio.io/trust-domain: ${cluster2} # Uncomment for NodePort-based cross-cluster traffic # peering.solo.io/preferred-data-plane-service-type: NodePort labels: topology.istio.io/network: ${cluster2} topology.kubernetes.io/region: "<cluster2_region>" # Uncomment as needed # topology.kubernetes.io/zone: "<cluster2_zone>" name: istio-remote-peer-${cluster2} namespace: istio-eastwest spec: addresses: # Change to 'type: Hostname' as needed, such as for AWS hostnames - type: IPAddress value: $CLUSTER2_EW_ADDRESS gatewayClassName: istio-remote listeners: # Comment HBONE out for NodePort-based cross-cluster traffic - name: cross-network port: 15008 protocol: HBONE tls: mode: Passthrough - name: xds-tls port: 15012 protocol: TLS tls: mode: Passthrough EOF- In the
Verify that peer linking was successful by running the
istioctl multicluster checkcommand. If any checks fail, run the command with--verbose, and see Validate your multicluster setup.istioctl multicluster check --contexts="$context1,$context2"In this example output, the remote peer gateways are successfully connected, and all other checks passed successfully. No global services exist because no app services are exposed across clusters in the multicluster mesh yet.
=== Cluster: cluster1 === ✅ Incompatible Environment Variable Check: all relevant environment variables are valid ✅ License Check: license is valid for multicluster ✅ CNI DNS Capture Check: AMBIENT_DNS_CAPTURE is enabled ✅ Pod Check (istiod): all pods healthy ✅ Pod Check (ztunnel): all pods healthy ✅ Pod Check (eastwest gateway istio-eastwest/istio-eastwest): all pods healthy ✅ Gateway Check: all eastwest gateways programmed ✅ istio-eastwest/istio-eastwest available at aab8471c7fcfa4a3c82f2d217b015d97-396238517.us-east-1.elb.amazonaws.com ✅ Peers Check: all clusters connected ✅ Connected to gloo-gateway-docs-mgt via ab46aa29a49914da789a6d5422aca279-541415195.us-east-2.elb.amazonaws.com ℹ️ Shared Services Check: no globally shared services found ====== === Cluster: cluster2 === ✅ Incompatible Environment Variable Check: all relevant environment variables are valid ✅ License Check: license is valid for multicluster ✅ CNI DNS Capture Check: AMBIENT_DNS_CAPTURE is enabled ✅ Pod Check (istiod): all pods healthy ✅ Pod Check (ztunnel): all pods healthy ✅ Pod Check (eastwest gateway istio-eastwest/istio-eastwest): all pods healthy ✅ Gateway Check: all eastwest gateways programmed ✅ istio-eastwest/istio-eastwest available at ab46aa29a49914da789a6d5422aca279-541415195.us-east-2.elb.amazonaws.com ✅ Peers Check: all clusters connected ✅ Connected to gloo-mesh-core-docs-mgt via aab8471c7fcfa4a3c82f2d217b015d97-396238517.us-east-1.elb.amazonaws.com ℹ️ Shared Services Check: no globally shared services found ====== ✅ Intermediate Certs Compatibility Check: all clusters have compatible intermediate certificates ✅ Network Configuration Check: all network configurations are valid ✅ Stale Workloads Check: skipped (flat network not detected)Optional: Verify that the istiod control plane for each peered cluster is included in each cluster’s proxy status list.
istioctl proxy-status --context ${context1} istioctl proxy-status --context ${context2}Example output for
cluster1, in which you can verify that the istiod control plane forcluster2is listed:NAME CLUSTER ISTIOD VERSION SUBSCRIBED TYPES istio-eastwest-67fd5679dc-fhsxs.istio-eastwest cluster1 istiod-7b7c9cc4c6-bdm9c 1.28.5-solo-fips 2 (WADS,WDS) istiod-6bc6765484-5bbhd.istio-system cluster2 istiod-7b7c9cc4c6-bdm9c 1.28.5-solo-fips 3 (FSDS,SGDS,WDS) ztunnel-5f8rb.kube-system cluster1 istiod-7b7c9cc4c6-bdm9c 1.28.5-solo-fips 2 (WADS,WDS) ztunnel-f96kh.kube-system cluster1 istiod-7b7c9cc4c6-bdm9c 1.28.5-solo-fips 2 (WADS,WDS) ztunnel-vtj4f.kube-system cluster1 istiod-7b7c9cc4c6-bdm9c 1.28.5-solo-fips 2 (WADS,WDS)
Next: Add apps to the ambient mesh. For multicluster setups, this includes making specific services available across your linked cluster setup.
Option 3: Automatically link clusters (beta)
In each cluster, use Helm to create the ambient mesh components, and create an east-west gateway so that traffic requests can be routed cross-cluster. Then, use the Gloo management plane to automate multicluster linking, which enables cross-cluster service discovery.
Review the following considerations:
- Multicluster mesh capabilities require an Enterprise level license for Solo Enterprise for Istio. If you do not have one, [contact an account representative](https://www.solo.io/company/contact).
- Automated peering requires Istio to be installed in the same cluster that the Gloo management plane is deployed to.
Set up tools
Set your Enterprise level license for Solo Enterprise for Istio as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing. Note that you might have previously saved this key in another variable, such as
${SOLO_LICENSE_KEY}or${GLOO_MESH_LICENSE_KEY}.export SOLO_ISTIO_LICENSE_KEY=<enterprise_license_key>Choose the version of Istio that you want to install or upgrade to by reviewing the supported versions.
Save the Solo distribution of Istio version.
export ISTIO_VERSION=1.28.5 export ISTIO_IMAGE=${ISTIO_VERSION}-soloSave the repo key for the minor version of the Solo distribution of Istio that you want to install. This is the 12-character hash at the end of the repo URL
us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Istio images built by Solo.io support article.# 12-character hash at the end of the repo URL export REPO_KEY=<repo_key> export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY} export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}Get the Solo distribution of Istio binary and install
istioctl, which you use for multicluster linking and gateway commands. This script automatically detects your OS and architecture, downloads the appropriate Solo distribution of Istio binary, and verifies the installation.bash <(curl -sSfL https://raw.githubusercontent.com/solo-io/doc-examples/main/istio/install-istioctl.sh) export PATH=${HOME}/.istioctl/bin:${PATH}Save the names and kubeconfig contexts of each cluster. This guide uses two clusters as an example. To add more clusters to the multicluster setup, include them in the arrays.
export cluster1=<cluster1_name> export context1=<cluster1_context> export cluster2=<cluster2_name> export context2=<cluster2_context>
Enable automatic peering of clusters
Upgrade Solo Enterprise for Istio in your multicluster setup to enable the ConfigDistribution feature flag and install the enterprise CRDs, which are required for Solo Enterprise for Istio to automate peering and distribute gateways between clusters.
In the cluster where you installed the management plane, upgrade your
gloo-platform-crdsHelm release to include the following settings.helm get values gloo-platform-crds -n gloo-mesh -o yaml --kube-context ${context1} > mgmt-crds.yaml helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \ --kube-context ${context1} \ --namespace gloo-mesh \ -f mgmt-crds.yaml \ --set featureGates.ConfigDistribution=true \ --set installEnterpriseCrds=trueIn the cluster where you installed the management plane, upgrade your
gloo-platformHelm release to include the following settings.helm get values gloo-platform -n gloo-mesh -o yaml --kube-context ${context1} > mgmt-plane.yaml helm upgrade gloo-platform gloo-platform/gloo-platform \ --kube-context ${context1} \ --namespace gloo-mesh \ -f mgmt-plane.yaml \ --set featureGates.ConfigDistribution=trueIn each connected cluster, upgrade your
gloo-platform-crdsHelm release to include the following settings. Repeat this step for each workload cluster.helm get values gloo-platform-crds -n gloo-mesh -o yaml --kube-context ${context2} > crds.yaml helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \ --kube-context ${context2} \ --namespace gloo-mesh \ -f crds.yaml \ --set installEnterpriseCrds=true
Create a shared root of trust
Each cluster in the multicluster setup must have a shared root of trust. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.
By default, the Istio CA generates a self-signed root certificate and key, and uses them to sign the workload certificates. For more information, see the Plug in CA Certificates guide in the community Istio documentation.
For demo installations, you can run the following function to quickly generate and plug in the certificates and key for the Istio CA:
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}
mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca
function create_cacerts_secret() {
context=${1:?context}
cluster=${2:?cluster}
make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
kubectl --context=${context} create ns istio-system || true
kubectl --context=${context} create secret generic cacerts -n istio-system \
--from-file=${cluster}/ca-cert.pem \
--from-file=${cluster}/ca-key.pem \
--from-file=${cluster}/root-cert.pem \
--from-file=${cluster}/cert-chain.pem
}
create_cacerts_secret ${context1} ${cluster1}
create_cacerts_secret ${context2} ${cluster2}
cd ../..To enhance the security of your setup even further and have full control over the Istio CA lifecycle, you can generate and store the root and intermediate CA certificates and keys with your own PKI provider. You can then use tools such as cert-manager to send certificate signing requests on behalf of istiod to your PKI provider. Cert-manager stores the signed intermediate certificates and keys in the cacerts Kubernetes secret so that istiod can use these credentials to issue leaf certificates for the workloads in the service mesh. You can set up cert-manager to also check the certificates and renew them before they expire.
AWS Private CA issuer and cert-manager: For an architectural overview of this certificate setup, see Bring your own Istio CAs with AWS. For steps on how to deploy this certificate setup, check out this Solo.io blog post. Be sure to repeat the steps so that a cacerts secret exists in each cluster.
Deploy ambient components
Apply the CRDs for the Kubernetes Gateway API to your cluster, which are required to create components such as waypoint proxies for L7 traffic policies, gateways with the
Gatewayresource, and more.kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml --context ${context1} kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml --context ${context2}Install the
basechart, which contains the CRDs and cluster roles required to set up Istio, in both clusters.function install_base() { context=${1:?context} helm upgrade --install istio-base oci://${HELM_REPO}/base \ --namespace istio-system \ --create-namespace \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF defaultRevision: "" profile: ambient EOF } install_base ${context1} install_base ${context2}function install_base() { context=${1:?context} helm upgrade --install istio-base oci://${HELM_REPO}/base \ --namespace istio-system \ --create-namespace \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF defaultRevision: "" profile: ambient global: platform: openshift EOF } install_base ${context1} install_base ${context2}You can optionally verify that the CRDs are successfully installed in both clusters.
kubectl get crds -l app.kubernetes.io/instance=istio-base --kube-context ${context1} kubectl get crds -l app.kubernetes.io/instance=istio-base --kube-context ${context2}Example output:
NAME CREATED AT authorizationpolicies.security.istio.io 2025-12-16T22:56:00Z destinationrules.networking.istio.io 2025-12-16T22:56:00Z envoyfilters.networking.istio.io 2025-12-16T22:56:00Z gateways.networking.istio.io 2025-12-16T22:56:00Z peerauthentications.security.istio.io 2025-12-16T22:56:00Z proxyconfigs.networking.istio.io 2025-12-16T22:56:00Z requestauthentications.security.istio.io 2025-12-16T22:56:00Z segments.admin.solo.io 2025-12-16T22:56:00Z serviceentries.networking.istio.io 2025-12-16T22:56:00Z sidecars.networking.istio.io 2025-12-16T22:56:00Z telemetries.telemetry.istio.io 2025-12-16T22:56:00Z virtualservices.networking.istio.io 2025-12-16T22:56:00Z wasmplugins.extensions.istio.io 2025-12-16T22:56:00Z workloadentries.networking.istio.io 2025-12-16T22:56:00Z workloadgroups.networking.istio.io 2025-12-16T22:56:00ZCreate the
istiodcontrol plane in both clusters.function install_istiod() { context=${1:?context} cluster=${2:?cluster} helm upgrade --install istiod oci://${HELM_REPO}/istiod \ --namespace istio-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF env: # Enables automatic creation of remote peer gateways PEERING_AUTOMATIC_LOCAL_GATEWAY: "true" # Assigns IP addresses to multicluster services PILOT_ENABLE_IP_AUTOALLOCATE: "true" # Disable community Istio multicluster mechanisms DISABLE_LEGACY_MULTICLUSTER: "true" # Required when meshConfig.trustDomain is set PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true" global: hub: ${REPO} multiCluster: clusterName: ${cluster} network: ${cluster} proxy: clusterDomain: cluster.local tag: ${ISTIO_IMAGE} meshConfig: accessLogFile: /dev/stdout defaultConfig: proxyMetadata: ISTIO_META_DNS_CAPTURE: "true" # Assign each cluster a unique trust domain to apply policies to specific clusters trustDomain: "${cluster}.local" pilot: cni: namespace: istio-system enabled: true # Required to enable multicluster support platforms: peering: enabled: true profile: ambient license: value: ${SOLO_ISTIO_LICENSE_KEY} # Uncomment if you prefer to specify your license secret # instead of an inline value. # secretRef: # name: # namespace: EOF } install_istiod ${context1} ${cluster1} install_istiod ${context2} ${cluster2}function install_istiod() { context=${1:?context} cluster=${2:?cluster} helm upgrade --install istiod oci://${HELM_REPO}/istiod \ --namespace istio-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF env: # Enables automatic creation of remote peer gateways PEERING_AUTOMATIC_LOCAL_GATEWAY: "true" # Assigns IP addresses to multicluster services PILOT_ENABLE_IP_AUTOALLOCATE: "true" # Disable community Istio multicluster mechanisms DISABLE_LEGACY_MULTICLUSTER: "true" # Required when meshConfig.trustDomain is set PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true" global: hub: ${REPO} multiCluster: clusterName: ${cluster} network: ${cluster} platform: openshift proxy: clusterDomain: cluster.local tag: ${ISTIO_IMAGE} meshConfig: accessLogFile: /dev/stdout defaultConfig: proxyMetadata: ISTIO_META_DNS_CAPTURE: "true" # Assign each cluster a unique trust domain to apply policies to specific clusters trustDomain: "${cluster}.local" pilot: cni: namespace: kube-system enabled: true # Required to enable multicluster support platforms: peering: enabled: true profile: ambient license: value: ${SOLO_ISTIO_LICENSE_KEY} # Uncomment if you prefer to specify your license secret # instead of an inline value. # secretRef: # name: # namespace: EOF } install_istiod ${context1} ${cluster1} install_istiod ${context2} ${cluster2}Install the Istio CNI node agent daemonset in both clusters. Note that although the CNI is included in this section, it is technically not part of the control plane or data plane.
function install_cni() { context=${1:?context} helm upgrade --install istio-cni oci://${HELM_REPO}/cni \ --namespace istio-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF # Assigns IP addresses to multicluster services ambient: dnsCapture: true excludeNamespaces: - istio-system - kube-system global: hub: ${REPO} tag: ${ISTIO_IMAGE} profile: ambient EOF } install_cni ${context1} install_cni ${context2}function install_cni() { context=${1:?context} helm upgrade --install istio-cni oci://${HELM_REPO}/cni \ --namespace kube-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF # Assigns IP addresses to multicluster services ambient: dnsCapture: true excludeNamespaces: - istio-system - kube-system global: hub: ${REPO} platform: openshift tag: ${ISTIO_IMAGE} profile: ambient EOF } install_cni ${context1} install_cni ${context2}Verify that the components of the Istio ambient control plane are successfully installed in both clusters. Because the Istio CNI is deployed as a daemon set, the number of CNI pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.
kubectl get pods -A --context ${context1} | grep istio kubectl get pods -A --context ${context2} | grep istioExample output:
istio-system istiod-85c4dfd97f-mncj5 1/1 Running 0 40s istio-system istio-cni-node-pr5rl 1/1 Running 0 9s istio-system istio-cni-node-pvmx2 1/1 Running 0 9s istio-system istio-cni-node-6q26l 1/1 Running 0 9sOptional: Check the istiod logs in both clusters to verify that the certificate you generated earlier is picked up by istiod.
kubectl logs deploy/istiod -n istio-system --context ${context1} | grep x509 kubectl logs deploy/istiod -n istio-system --context ${context2} | grep x509Example output:
2025-12-16T22:59:06.783901Z info x509 cert - Issuer: "CN=Intermediate CA,O=Istio,L=cluster-1", Subject: "", SN: def320623729b8370172413749143836, NotBefore: "2025-12-16T22:57:06Z", NotAfter: "2035-12-14T22:59:06Z" 2025-12-16T22:59:06.783937Z info x509 cert - Issuer: "CN=Root CA,O=Istio", Subject: "CN=Intermediate CA,O=Istio,L=cluster-1", SN: 452d45254328667ccf8434c64c79fe789612bb5a, NotBefore: "2025-12-16T22:54:30Z", NotAfter: "2035-12-14T22:54:30Z" 2025-12-16T22:59:06.783966Z info x509 cert - Issuer: "CN=Root CA,O=Istio", Subject: "CN=Root CA,O=Istio", SN: 38dad68e56ab8c506ae07454801f65b134bd9580, NotBefore: "2025-12-16T22:54:30Z", NotAfter: "2035-12-14T22:54:30Z"Install the ztunnel daemonset in both clusters.
function install_ztunnel() { context=${1:?context} cluster=${2:?cluster} helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \ --namespace istio-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF configValidation: true enabled: true env: L7_ENABLED: "true" # Required when a unique trust domain is set for each cluster SKIP_VALIDATE_TRUST_DOMAIN: "true" hub: ${REPO} istioNamespace: istio-system multiCluster: clusterName: ${cluster} namespace: istio-system network: ${cluster} profile: ambient proxy: clusterDomain: cluster.local tag: ${ISTIO_IMAGE} terminationGracePeriodSeconds: 29 variant: distroless EOF } install_ztunnel ${context1} ${cluster1} install_ztunnel ${context2} ${cluster2}function install_ztunnel() { context=${1:?context} cluster=${2:?cluster} helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \ --namespace kube-system \ --kube-context ${context} \ --version ${ISTIO_IMAGE} \ -f - <<EOF configValidation: true enabled: true env: L7_ENABLED: "true" # Required when a unique trust domain is set for each cluster SKIP_VALIDATE_TRUST_DOMAIN: "true" global: platform: openshift hub: ${REPO} istioNamespace: istio-system multiCluster: clusterName: ${cluster} namespace: kube-system network: ${cluster} profile: ambient proxy: clusterDomain: cluster.local tag: ${ISTIO_IMAGE} terminationGracePeriodSeconds: 29 variant: distroless EOF } install_ztunnel ${context1} ${cluster1} install_ztunnel ${context2} ${cluster2}Verify that the ztunnel pods are successfully installed in both clusters. Because the ztunnel is deployed as a daemon set, the number of pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.
kubectl get pods -A --context ${context1} | grep ztunnel kubectl get pods -A --context ${context2} | grep ztunnelExample output:
ztunnel-tvtzn 1/1 Running 0 7s ztunnel-vtpjm 1/1 Running 0 4s ztunnel-hllxg 1/1 Running 0 4sLabel the
istio-systemnamespace with the clusters’ network names, which you previously set to each cluster name in theglobal.networkfield of theistiodinstallations. The ambient control plane uses this label internally to group pods that exist in the same L3 network.kubectl label namespace istio-system --context ${context1} topology.istio.io/network=${cluster1} kubectl label namespace istio-system --context ${context2} topology.istio.io/network=${cluster2}Create an east-west gateway in the
istio-eastwestnamespace. In each cluster, the east-west gateway is implemented as a ztunnel that facilitates traffic between services across clusters in your multicluster mesh.You can use either LoadBalancer or NodePort addresses for cross-cluster traffic.
Create the east-west gateway in both clusters. Cross-cluster traffic though this gateway resolves to the LoadBalancer address. For customization options, see the gateway guide in the Istio docs.
- Solo distribution of
istioctl: For more information about this command, see the CLI reference.function create_ew_gateway_lb() { context=${1:?context} cluster=${2:?cluster} kubectl create namespace istio-eastwest --context ${context} istioctl multicluster expose --namespace istio-eastwest --context ${context} --generate > ew-gateway-${cluster}.yaml kubectl apply -f ew-gateway-${cluster}.yaml --context ${context} } create_ew_gateway_lb ${context1} ${cluster1} create_ew_gateway_lb ${context2} ${cluster2}
Create the east-west gateway in both clusters. Note that the gateway must still be created with a stable internal IP address, which is required for xDS communication with the istiod control plane in each cluster. NodePort peering is used for data plane communication, in that requests to services resolve to the NodePort instead of the gateway’s stable IP address. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.
- Solo distribution of
istioctl: For more information about this command, see the CLI reference.function create_ew_gateway_nodeport() { context=${1:?context} cluster=${2:?cluster} kubectl create namespace istio-eastwest --context ${context} istioctl multicluster expose --namespace istio-eastwest --context ${context} --generate > ew-gateway-${cluster}.yaml kubectl apply -f ew-gateway-${cluster}.yaml --context ${context} kubectl annotate gateway istio-eastwest -n istio-eastwest peering.solo.io/data-plane-service-type=NodePort --context ${context} } create_ew_gateway_nodeport ${context1} ${cluster1} create_ew_gateway_nodeport ${context2} ${cluster2}
Verify that the east-west gateway is successfully deployed in both clusters.
kubectl get svc -n istio-eastwest --context ${context1} kubectl get svc -n istio-eastwest --context ${context2}Example output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE istio-eastwest LoadBalancer 172.20.205.104 <external_address> 15021:31655/TCP,15008:32699/TCP,15012:32166/TCP 55s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE istio-eastwest LoadBalancer 172.20.21.117 <external_address> 15021:30324/TCP,15008:31875/TCP,15012:31050/TCP 77s
Review remote peer gateways
After you complete the steps for each cluster, verify that Solo Enterprise for Istio successfully created and distributed the remote peering gateways. These gateways use the
istio-remoteGatewayClass, which allows the istiod control plane in each cluster to discover the east-west gateway addresses of other clusters. Solo Enterprise for Istio generates oneistio-remoteresource in the cluster where the management plane is deployed for each connected cluster, and then distributes the gateway to each cluster respectively.Verify that an
istio-remotegateway for each connected cluster is copied to the cluster where the management plane is deployed.kubectl get gateways -n istio-eastwest --context ${context1}In this example output, the
istio-remotegateway that was auto-generated for connected clustercluster2is copied tocluster1where the management plane is deployed, alongsidecluster1’s ownistio-remotegateway and east-west gateway.NAMESPACE NAME CLASS ADDRESS PROGRAMMED AGE istio-eastwest istio-eastwest istio-eastwest a7f6f1a2611fc4eb3864f8d688622fd4-1234567890.us-east-1.elb.amazonaws.com True 6s istio-eastwest istio-remote-peer-cluster1 istio-remote a5082fe9522834b8192a6513eb8c6b01-0987654321.us-east-1.elb.amazonaws.com True 4s istio-eastwest istio-remote-peer-cluster2 istio-remote aaad62dc3ffb142a1bfc13df7fe9665b-5678901234.us-east-1.elb.amazonaws.com True 4sIn each connected cluster, verify that all
istio-remotegateways are successfully distributed to all workload clusters. This ensures that services in each workload cluster can now access the east-west gateways in other clusters of the multicluster mesh setup.kubectl get gateways -n istio-eastwest --context ${context2}Verify that peer linking was successful by running the
istioctl multicluster checkcommand. For more information about this command, see the CLI reference. If any checks fail, run the command with--verbose, and see Validate your multicluster setup.istioctl multicluster check --contexts="$context1,$context2"Example output:
=== Cluster: cluster1 === ✅ Incompatible Environment Variable Check: all relevant environment variables are valid ✅ License Check: license is valid for multicluster ✅ CNI DNS Capture Check: AMBIENT_DNS_CAPTURE is enabled ✅ Pod Check (istiod): all pods healthy ✅ Pod Check (ztunnel): all pods healthy ✅ Pod Check (eastwest gateway istio-eastwest/istio-eastwest): all pods healthy ✅ Gateway Check: all eastwest gateways programmed ✅ istio-eastwest/istio-eastwest available at aab8471c7fcfa4a3c82f2d217b015d97-396238517.us-east-1.elb.amazonaws.com ✅ Peers Check: all clusters connected ✅ Connected to gloo-gateway-docs-mgt via ab46aa29a49914da789a6d5422aca279-541415195.us-east-2.elb.amazonaws.com ℹ️ Shared Services Check: no globally shared services found ========= Cluster: cluster2 === ✅ Incompatible Environment Variable Check: all relevant environment variables are valid ✅ License Check: license is valid for multicluster ✅ CNI DNS Capture Check: AMBIENT_DNS_CAPTURE is enabled ✅ Pod Check (istiod): all pods healthy ✅ Pod Check (ztunnel): all pods healthy ✅ Pod Check (eastwest gateway istio-eastwest/istio-eastwest): all pods healthy ✅ Gateway Check: all eastwest gateways programmed ✅ istio-eastwest/istio-eastwest available at ab46aa29a49914da789a6d5422aca279-541415195.us-east-2.elb.amazonaws.com ✅ Peers Check: all clusters connected ✅ Connected to gloo-mesh-core-docs-mgt via aab8471c7fcfa4a3c82f2d217b015d97-396238517.us-east-1.elb.amazonaws.com ℹ️ Shared Services Check: no globally shared services found
✅ Intermediate Certs Compatibility Check: all clusters have compatible intermediate certificates ✅ Network Configuration Check: all network configurations are valid ✅ Stale Workloads Check: skipped (flat network not detected)
<button class=“hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50” title=“Copy code” aria-label=“Copy code” data-copied-label=“Copied!”<div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"></div>
Next
- Add apps to the ambient mesh. For multicluster setups, this includes making specific services available across your linked cluster setup.
- In a multicluster mesh, the east-west gateway serves as a ztunnel that allows traffic requests to flow across clusters, but it does not modify requests in any way. To control in-mesh traffic, you can instead apply policies to waypoint proxies that you create for a workload namespace.
Optional: Validate your multicluster setup
Both before and after you link clusters into a multicluster mesh, you can use theistioctl multicluster checkcommand, along with other observability checks, to verify multiple aspects of multicluster ambient mesh support and status.istioctl multicluster check
You can use the
istioctl multicluster check --precheckcommand to check the individual readiness of each cluster before runningistioctl multicluster linkto link them in a multicluster mesh, and run it again after linking to confirm that the connections were successful. This command performs checks listed in the following sections, which you can review to understand what each check validates. Additionally, if any of the checks fail, run the command with the--verboseoption, and review the following troubleshooting recommendations.istioctl multicluster check --verbose --contexts="$context1,$context2"For more information about this command, see the CLI reference.
Incompatible environment variables
Checks whether the
ENABLE_PEERING_DISCOVERY=trueand optionallyK8S_SELECT_WORKLOAD_ENTRIES=trueenvironment variables are set incorrectly or are not supported for multicluster ambient mesh.Example verbose output:
--- Incompatible Environment Variable Check --- ✅ Incompatible Environment Variable Check: K8S_SELECT_WORKLOAD_ENTRIES is valid ("") ✅ Incompatible Environment Variable Check: ENABLE_PEERING_DISCOVERY is valid ("true") ✅ Incompatible Environment Variable Check: all relevant environment variables are validIf this check fails, check your environment variables in your istiod configuration, such as by running
helm get values --kube-context ${CLUSTER_CONTEXT} istiod -n istio-system -o yaml, and update your configuration.License validity
Checks whether the license in use by istiod is valid for multicluster ambient mesh. Multicluster capabilities require an Enterprise level license for Solo Enterprise for Istio.
Example verbose output:
--- License Check --- ✅ License Check: license is valid for multiclusterIf your license does not support multicluster ambient mesh, contact your Solo account representative.
Pod health
Checks the health of the pods in the cluster. All istiod, ztunnel, and east-west gateway pods across the checked clusters must be healthy and running for the multicluster mesh to function correctly.
Example verbose output:
--- Pod Check (istiod) --- NAME READY STATUS RESTARTS AGE istiod-6d9cdf88cf-l47tf 1/1 Running 0 10m18s ✅ Pod Check (istiod): all pods healthy --- Pod Check (ztunnel) --- NAME READY STATUS RESTARTS AGE ztunnel-dvlwk 1/1 Running 0 10m6s ✅ Pod Check (ztunnel): all pods healthy --- Pod Check (eastwest gateway) --- NAME READY STATUS RESTARTS AGE istio-eastwest-857b77fc5d-qgnrl 1/1 Running 0 9m33s ✅ Pod Check (eastwest gateway): all pods healthyTo check any unhealthy pods, run the following commands. Consider checking the pod logs, and review Debug Istio.
kubectl get po -n istio-system kubectl get po -n istio-eastwestEast-west gateway status
Checks the status of the east-west gateways in the cluster. When an east-west gateway is created, the gateway controller creates a Kubernetes service to expose the gateway. Once this service is correctly attached to the gateway and has an address assigned, the east-west gateway has a
Programmedstatus oftrue.Example verbose output:
--- Gateway Check --- Gateway: istio-eastwest Addresses: - 172.18.7.110 Status: programmed ✅ ✅ Gateway Check: all eastwest gateways programmedIf the
Programmedstatus is nottrue, an issue might exist with the address allocation for the service. Check the east-west gateway with a command such askubectl get svc -n istio-eastwest, and verify that your cloud provider can correctly allocate addresses to the service.Remote peer gateway status
Checks the status of the remote peer gateways in the cluster, which represent the other peered clusters in the multicluster setup. These remote gateways configure the connection between the local cluster’s istiod control plane, and the peered clusters’ remote networks to enable xDS communication between peers. When the initial network connection between istiod and a remote peer is made, the gateway’s
gloo.solo.io/PeerConnectedstatus updates totrue. Then, when the full xDS sync occurs between peers, the gateway’sgloo.solo.io/PeeringSucceededstatus also updates totrue. This check ensures that both statuses aretrue.Example verbose output:
--- Peers Check --- Cluster: cluster2 Addresses: - 172.18.7.130 Conditions: - Accepted: True - Programmed: True - gloo.solo.io/PeerConnected: True - gloo.solo.io/PeeringSucceeded: True - gloo.solo.io/PeerDataPlaneProgrammed: True Status: connected ✅ ✅ Peers Check: all clusters connectedIf the connection is severed between the peers, the
gloo.solo.io/PeerConnectedstatus becomesfalse. A failed connection between peers can be due to either a misconfiguration in the peering setup, or a network issue blocking port15008on the remote cluster, which is the cross-network HBONE port that the east-west gateway listens on. Review the steps you took to link clusters together, such as the steps outlined in the Helm default network guide. Additionally, review any firewall rules or network policies that might block access through port15008on the remote cluster.Intermediate certificate compatibility
Confirms the certificate compatibility between peered clusters. This check reads the
root-cert.pemfrom theistio-ca-root-certconfigmap in theistio-systemnamespace, and usesx509certificate validation to confirm the root cert is compatible with all of the clusters’ca-cert.pemintermediate certificate chains from thecacertssecret.Example verbose output:
--- Intermediate Certs Compatibility Check --- ℹ Intermediate Certs Compatibility Check: cluster cluster1 root certificate SHA256 sum: 6d18f32e134824c158d97f32618657c45d5a83839f838ada751757139481537e ℹ Intermediate Certs Compatibility Check: cluster cluster2 root certificate SHA256 sum: 6d18f32e134824c158d97f32618657c45d5a83839f838ada751757139481537e ✅ Intermediate Certs Compatibility Check: cluster cluster1 has compatible intermediate certificates with cluster cluster2 ✅ Intermediate Certs Compatibility Check: cluster cluster2 has compatible intermediate certificates with cluster cluster1 ✅ Intermediate Certs Compatibility Check: all clusters have compatible intermediate certificatesIf this check fails because the root certs are not valid for each peered clusters’ intermediate certificate chain, you can check the istiod logs for TLS errors when attempting to communicate with a peered cluster, such as the following:
2025-12-04T22:09:22.474517Z warn deltaadsc disconnected, retrying in 24.735483751s: delta stream: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: remote error: tls: unknown certificate authority" target=peering-cluster2Ensure each cluster has a
cacertssecret in theistio-systemnamespace. To regenerate invalid certificates for each cluster, follow the example steps in Create a shared root of trust.Network configuration
Confirms the network configuration of the multicluster mesh. For multicluster peering setups that do not use a flat network topology, each cluster must occupy a unique network. The network name must be defined with the label
topology.istio.io/networkand set on both theistio-systemnamespace and theistio-eastwestgateway resource. The same network name must also be set as theNETWORKenvironment variable on the ztunnel daemonset. Each remote gateway that represents that cluster must have thetopology.istio.io/networklabel equal to the network of the remote cluster.Example verbose output:
--- Network Configuration Check --- ✅ Cluster cluster1 has network: cluster1 ✅ Eastwest gateway istio-eastwest/istio-eastwest has correct network label: cluster1 ✅ Cluster cluster2 has network: cluster2 ✅ Eastwest gateway istio-eastwest/istio-eastwest has correct network label: cluster2 ✅ Remote gateway istio-eastwest/istio-remote-peer-cluster2 references network cluster2 (clusters: [cluster2]) ✅ Remote gateway istio-eastwest/istio-remote-peer-cluster1 references network cluster1 (clusters: [cluster1]) ✅ Network Configuration Check: all network configurations are validMismatched network identities cause errors in cross-cluster communication, which leads to error logs in ztunnel pods that indicate a network timeout on the outbound communication. Notably, the destination address on these errors is a
240.X.X.Xaddress, instead of the correct remote peer gateway address. You can runkubectl logs -l app=ztunnel -n istio-system --tail=10 --context ${CLUSTER_CONTEXT} | grep -iE "error|warn"to review logs such as the following:2025-11-18T16:14:53.490573Z error access connection complete src.addr=240.0.2.27:46802 src.workload="ratings-v1-5dc79b6bcd-zm8v6" src.namespace="bookinfo" src.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings" dst.addr=240.0.9.43:15008 dst.hbone_addr=240.0.9.43:9080 dst.service="productpage.bookinfo.mesh.internal" dst.workload="autogenflat.portfolio1-soloiopoc-cluster1.bookinfo.productpage-v1-54bb874995-hblwp.ee508601917c" dst.namespace="bookinfo" dst.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-productpage" direction="outbound" bytes_sent=0 bytes_recv=0 duration="10001ms" error="connection timed out, maybe a NetworkPolicy is blocking HBONE port 15008: deadline has elapsed"To troubleshoot these issues, be sure that you use unique network names to represent each cluster, and that you correctly labeled the cluster’s
istio-systemnamespace with that network name, such as by runningkubectl label namespace istio-system --context ${CLUSTER_CONTEXT} topology.istio.io/network=${CLUSTER_NAME}. You can also relabel the east-west gateway in the cluster, and the remote peer gateways in other clusters that represent this cluster.Stale workload entries
In flat network setups, checks for any outdated workload entries that must be removed from the multicluster mesh. Stale workload entries might exist from pods that were deleted, but the autogenerated entries for those workloads were not correctly cleaned up. If you do not use a flat network topology, no autogenerated workload entries exist to be validated, and this check can be ignored.
Example verbose output for a non-flat network setup:
--- Stale Workloads Check --- ⚠ Stale Workloads Check: no autogenflat workload entries foundIf you use a flat network topology, and this check fails with stale workload entries, run
kubectl get workloadentries -n istio-system | grep autogenflatto list the autogenerated workload entries in the remote cluster, and compare the list to the output ofkubectl get podsin the source cluster for those workloads. You can safely manually delete the stale workload entries in the remote cluster for pods that no longer exist in the source cluster, such as by runningkubectl get workloadentries -n istio-system <entry_name>.Further debugging and observability
For additional guidance around observing your multicluster ambient mesh, check out the observability overview, which contains links to guides on using logs, metrics, and traces in your Istio environment.
For additional guidance around debugging your multicluster ambient mesh, check out the Istio troubleshooting guide.