Skip to content
home
2.11.x
Istio ambient mesh
Multicluster
Install
You are viewing the documentation for Solo Enterprise for Istio, formerly known as Gloo Mesh (OSS APIs).

Flat networking (advanced)

Page as Markdown
Enable multicluster service discovery and mesh communication by using Istio’s flat network peering capability.
About flat networking

In a traditional multicluster setup, you typically use different networks for each cluster. To connect these different networks and route traffic across clusters, you need to use an east-west gateway.
In a flat network setup, also referred to as single network, all your clusters use the same network. Because of that, services can communicate with each other by using a service’s IP address directly. No east-west gateway is required to handle dataplane traffic. To avoid IP address conflicts, each cluster typically is assigned a unique IP CIDR range. You can establish a flat network by using a VPN, a CNI that supports the Border Gateway Protocol (BGP), or other overlay solutions for all of your clusters.
For Istio to presume a flat network, all your clusters must use the same network name in the topology.istio.io/network label and spec.values.global.network field. Istio then routes traffic by using a service’s IP address directly and does not send traffic through an east-west gateway.
About this guide

In this guide, you explore how to set up an Istio in a flat network multicluster setup. You complete the following tasks:
  • Set up two kind clusters that are connected by using a flat network.
  • Install the Solo distribution of Istio in each cluster by using the flat network topology and ambient profile. The ambient profile is required, even if you plan to run sidecar-injected Istio workloads in your cluster.
  • Peer the clusters by using east-west and peering gateways. These gateways are used to allow communication between istiod instances.
  • Expose a service globally across both clusters.
  • Test routing between clusters by using the service’s IP address directly instead of the east-west gateway.
Before you begin

  1. Set your Enterprise level license for Solo Enterprise for Istio as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing. Note that you might have previously saved this key in another variable, such as ${SOLO_LICENSE_KEY} or ${GLOO_MESH_LICENSE_KEY}.
    export SOLO_ISTIO_LICENSE_KEY=<enterprise_license_key>
  2. Choose the version of Istio that you want to install or upgrade to by reviewing the supported versions.
  3. Save the Solo distribution of Istio version.
    export ISTIO_VERSION=1.28.5
export ISTIO_IMAGE=${ISTIO_VERSION}-solo
```<ol start="4">
  • Save the repo key for the minor version of the Solo distribution of Istio that you want to install. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Istio images built by Solo.io support article.
    # 12-character hash at the end of the repo URL
export REPO_KEY=<repo_key>
export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}
    1. Get the Solo distribution of Istio binary and install istioctl, which you use for multicluster linking and gateway commands. This script automatically detects your OS and architecture, downloads the appropriate Solo distribution of Istio binary, and verifies the installation.
      bash <(curl -sSfL https://raw.githubusercontent.com/solo-io/doc-examples/main/istio/install-istioctl.sh)
export PATH=${HOME}/.istioctl/bin:${PATH}
    Step 1: Set up a flat network test environment

    Set up a kind cluster environment that consists of two kind clusters that are connected by using a flat network. To establish a flat network, you create these kind clusters with custom pod and service subnets, and connect these subnets with custom routes.
    1. Install the cloud-provider-kind CLI. This CLI allows you to assign an external IP address to a LoadBalancerIP service type. An external IP address is required for the east-west gateways to enable proper peering between multiple clusters.
      go install sigs.k8s.io/cloud-provider-kind@latest
    2. Start the cloud-provider-kind CLI.
      sudo cloud-provider-kind
    3. In a separate terminal window, download and run the setup-clusters.sh script. This script sets up the following components.
      • A kind cluster cluster-1 that uses the 10.10.0.0/16 CIDR for the pod subnet and the
10.255.10.0/24 CIDR for the service subnet.
      • Another kind cluster cluster-2 that uses the 10.20.0.0/16 CIDR for the pod subnet and the 10.255.20.0/24 CIDR for the service subnet.
      • Routes between cluster-1 and cluster-2 to allow communication between pods by using the pod’s IP address directly.
      curl -L https://raw.githubusercontent.com/solo-io/doc-examples/main/istio/flat-network/setup-clusters.sh -o setup-clusters.sh
chmod +x setup-clusters.sh
./setup-clusters.sh
    4. Save the name and context for both of your clusters in an environment variable.
      cluster1="cluster-1"
cluster2="cluster-2"
context1="kind-${cluster1}"
context2="kind-${cluster2}"
    Step 2: Install Istio

    Install the Solo distribution of Istio in your cluster.
    1. Create a shared root of trust.
      Each cluster in the multicluster setup must have a shared root of trust. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.
      By default, the Istio CA generates a self-signed root certificate and key, and uses them to sign the workload certificates. For more information, see the Plug in CA Certificates guide in the community Istio documentation.
      For demo installations, you can run the following function to quickly generate and plug in the certificates and key for the Istio CA:
      curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}

mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

function create_cacerts_secret() {
  context=${1:?context}
  cluster=${2:?cluster}
  make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
  kubectl --context=${context} create ns istio-system || true
  kubectl --context=${context} create secret generic cacerts -n istio-system \
    --from-file=${cluster}/ca-cert.pem \
    --from-file=${cluster}/ca-key.pem \
    --from-file=${cluster}/root-cert.pem \
    --from-file=${cluster}/cert-chain.pem
}

create_cacerts_secret ${context1} ${cluster1}
create_cacerts_secret ${context2} ${cluster2}

cd ../..
      To enhance the security of your setup even further and have full control over the Istio CA lifecycle, you can generate and store the root and intermediate CA certificates and keys with your own PKI provider. You can then use tools such as cert-manager to send certificate signing requests on behalf of istiod to your PKI provider. Cert-manager stores the signed intermediate certificates and keys in the cacerts Kubernetes secret so that istiod can use these credentials to issue leaf certificates for the workloads in the service mesh. You can set up cert-manager to also check the certificates and renew them before they expire.
      AWS Private CA issuer and cert-manager: For an architectural overview of this certificate setup, see Bring your own Istio CAs with AWS. For steps on how to deploy this certificate setup, check out this Solo.io blog post. Be sure to repeat the steps so that a cacerts secret exists in each cluster.
    2. Install the Kubernetes Gateway API in both of your clusters. The API is required to spin up east-west gateways later.
      kubectl --context=${context1} apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml
kubectl --context=${context2} apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml
    3. Install the base chart, which contains the CRDs and cluster roles required to set up Istio, in both clusters.
      for context in ${context1} ${context2}; do
  helm upgrade --install istio-base oci://${HELM_REPO}/base \
  --namespace istio-system \
  --create-namespace \
  --kube-context $context \
  --version ${ISTIO_IMAGE} \
  -f - <<EOF
defaultRevision: ""
profile: ambient
EOF
done
      for context in ${context1} ${context2}; do
  helm upgrade --install istio-base oci://${HELM_REPO}/base \
  --namespace istio-system \
  --create-namespace \
  --kube-context $context \
  --version ${ISTIO_IMAGE} \
  -f - <<EOF
defaultRevision: ""
profile: ambient
global:
  platform: openshift
EOF
done
      You can optionally verify that the CRDs are successfully installed in both clusters.
      kubectl get crds -l app.kubernetes.io/instance=istio-base --kube-context ${context1}
kubectl get crds -l app.kubernetes.io/instance=istio-base --kube-context ${context2}
      Example output:
      NAME                                       CREATED AT
authorizationpolicies.security.istio.io    2025-12-16T22:56:00Z
destinationrules.networking.istio.io       2025-12-16T22:56:00Z
envoyfilters.networking.istio.io           2025-12-16T22:56:00Z
gateways.networking.istio.io               2025-12-16T22:56:00Z
peerauthentications.security.istio.io      2025-12-16T22:56:00Z
proxyconfigs.networking.istio.io           2025-12-16T22:56:00Z
requestauthentications.security.istio.io   2025-12-16T22:56:00Z
segments.admin.solo.io                     2025-12-16T22:56:00Z
serviceentries.networking.istio.io         2025-12-16T22:56:00Z
sidecars.networking.istio.io               2025-12-16T22:56:00Z
telemetries.telemetry.istio.io             2025-12-16T22:56:00Z
virtualservices.networking.istio.io        2025-12-16T22:56:00Z
wasmplugins.extensions.istio.io            2025-12-16T22:56:00Z
workloadentries.networking.istio.io        2025-12-16T22:56:00Z
workloadgroups.networking.istio.io         2025-12-16T22:56:00Z
    4. Create the istiod control plane in your cluster. Note that for Istio to presume a flat network, all clusters must use the same network name in the global.network field. The network name is an arbitrary value and can be set to a string of your choice. In this example, flat-network is used as the network name.
      helm upgrade --install istiod oci://${HELM_REPO}/istiod \
--namespace istio-system \
--kube-context ${context1} \
--version ${ISTIO_IMAGE} \
-f - <<EOF
env:
  PILOT_ENABLE_IP_AUTOALLOCATE: "true"
  PEERING_ENABLE_FLAT_NETWORKS: "true"<pre><code> DISABLE_LEGACY_MULTICLUSTER: &quot;true&quot;
</code></pre>

global:
  hub: ${REPO}
  multiCluster:
    clusterName: "${cluster1}"
  network: "flat-network"
  tag: ${ISTIO_IMAGE}
meshConfig:
  accessLogFile: /dev/stdout
  defaultConfig:
    proxyMetadata:
      ISTIO_META_DNS_AUTO_ALLOCATE: "true"
      ISTIO_META_DNS_CAPTURE: "true"
pilot:
  cni:
    namespace: istio-system
    enabled: true
platforms:
  peering:
    enabled: true
profile: ambient
license:
  value: ${SOLO_ISTIO_LICENSE_KEY}
EOF
      helm upgrade --install istiod oci://${HELM_REPO}/istiod \
--namespace istio-system \
--kube-context ${context2} \
--version ${ISTIO_IMAGE} \
-f - <<EOF
env:
  PILOT_ENABLE_IP_AUTOALLOCATE: "true"
  PEERING_ENABLE_FLAT_NETWORKS: "true"<pre><code> DISABLE_LEGACY_MULTICLUSTER: &quot;true&quot;
</code></pre>

global:
  hub: ${REPO}
  multiCluster:
    clusterName: "${cluster2}"
  network: "flat-network"
  tag: ${ISTIO_IMAGE}
meshConfig:
  accessLogFile: /dev/stdout
  defaultConfig:
    proxyMetadata:
      ISTIO_META_DNS_AUTO_ALLOCATE: "true"
      ISTIO_META_DNS_CAPTURE: "true"
pilot:
  cni:
    namespace: istio-system
    enabled: true
platforms:
  peering:
    enabled: true
profile: ambient
license:
  value: ${SOLO_ISTIO_LICENSE_KEY}
EOF
      helm upgrade --install istiod oci://${HELM_REPO}/istiod \
--namespace istio-system \
--kube-context ${context1} \
--version ${ISTIO_IMAGE} \
-f - <<EOF
env:
  PILOT_ENABLE_IP_AUTOALLOCATE: "true"
  PEERING_ENABLE_FLAT_NETWORKS: "true"<pre><code> DISABLE_LEGACY_MULTICLUSTER: &quot;true&quot;
</code></pre>

global:
  hub: ${REPO}
  multiCluster:
    clusterName: "${cluster1}"
  network: "flat-network"
  platform: openshift
  tag: ${ISTIO_IMAGE}
meshConfig:
  accessLogFile: /dev/stdout
  defaultConfig:
    proxyMetadata:
      ISTIO_META_DNS_AUTO_ALLOCATE: "true"
      ISTIO_META_DNS_CAPTURE: "true"
pilot:
  cni:
    namespace: istio-system
    enabled: true
platforms:
  peering:
    enabled: true
profile: ambient
license:
  value: ${SOLO_ISTIO_LICENSE_KEY}
EOF
      helm upgrade --install istiod oci://${HELM_REPO}/istiod \
--namespace istio-system \
--kube-context ${context2} \
--version ${ISTIO_IMAGE} \
-f - <<EOF
env:
  PILOT_ENABLE_IP_AUTOALLOCATE: "true"
  PEERING_ENABLE_FLAT_NETWORKS: "true"<pre><code> DISABLE_LEGACY_MULTICLUSTER: &quot;true&quot;
</code></pre>

global:
  hub: ${REPO}
  multiCluster:
    clusterName: "${cluster2}"
  network: "flat-network"
  platform: openshift
  tag: ${ISTIO_IMAGE}
meshConfig:
  accessLogFile: /dev/stdout
  defaultConfig:
    proxyMetadata:
      ISTIO_META_DNS_AUTO_ALLOCATE: "true"
      ISTIO_META_DNS_CAPTURE: "true"
pilot:
  cni:
    namespace: istio-system
    enabled: true
platforms:
  peering:
    enabled: true
profile: ambient
license:
  value: ${SOLO_ISTIO_LICENSE_KEY}
EOF
    5. Install the Istio CNI node agent daemonset in both clusters. Note that although the CNI is included in this section, it is technically not part of the control plane or data plane.
      for context in ${context1} ${context2}; do
  helm upgrade --install istio-cni oci://${HELM_REPO}/cni \
  --namespace istio-system \
  --kube-context $context \
  --version ${ISTIO_IMAGE} \
  -f - <<EOF
ambient:
  dnsCapture: true
excludeNamespaces:
  - istio-system
  - kube-system
global:
  hub: ${REPO}
  tag: ${ISTIO_IMAGE}
profile: ambient
EOF
done
      for context in ${context1} ${context2}; do
  helm upgrade --install istio-cni oci://${HELM_REPO}/cni \
  --namespace istio-system \
  --kube-context $context \
  --version ${ISTIO_IMAGE} \
  -f - <<EOF
ambient:
  dnsCapture: true
excludeNamespaces:
  - istio-system
  - kube-system
global:
  hub: ${REPO}
  tag: ${ISTIO_IMAGE}
  platform: openshift
profile: ambient
EOF
done
    6. Install the ztunnel daemonset. Make sure to use the same network name in the network field as you used in istiod’s global.network field. In this example, flat-network is used as the network name.
      helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
--namespace istio-system \
--kube-context ${context1} \
--version ${ISTIO_IMAGE} \
-f - <<EOF
configValidation: true
enabled: true
env:
  L7_ENABLED: "true"
  # Required when a unique trust domain is set for each cluster
hub: ${REPO}
istioNamespace: istio-system
multiCluster:
  clusterName: ${cluster1}
namespace: istio-system
network: "flat-network"
platforms:
  peering:
    enabled: true
profile: ambient
tag: ${ISTIO_IMAGE}
terminationGracePeriodSeconds: 29
variant: distroless
EOF
      helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
--namespace istio-system \
--kube-context ${context2} \
--version ${ISTIO_IMAGE} \
-f - <<EOF
configValidation: true
enabled: true
env:
  L7_ENABLED: "true"
  # Required when a unique trust domain is set for each cluster
hub: ${REPO}
istioNamespace: istio-system
multiCluster:
  clusterName: ${cluster2}
namespace: istio-system
network: "flat-network"
platforms:
  peering:
    enabled: true
profile: ambient
tag: ${ISTIO_IMAGE}
terminationGracePeriodSeconds: 29
variant: distroless
EOF
      helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
--namespace istio-system \
--kube-context ${context1} \
--version ${ISTIO_IMAGE} \
-f - <<EOF
configValidation: true
enabled: true
env:
  L7_ENABLED: "true"
global: 
  platform: openshift
hub: ${REPO}
istioNamespace: istio-system
multiCluster:
  clusterName: ${cluster1}
namespace: istio-system
network: "flat-network"
platforms:
  peering:
    enabled: true
profile: ambient
tag: ${ISTIO_IMAGE}
terminationGracePeriodSeconds: 29
variant: distroless
EOF
      helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
--namespace istio-system \
--kube-context ${context2} \
--version ${ISTIO_IMAGE} \
-f - <<EOF
configValidation: true
enabled: true
env:
  L7_ENABLED: "true"
global: 
  platform: openshift
hub: ${REPO}
istioNamespace: istio-system
multiCluster:
  clusterName: ${cluster2}
namespace: istio-system
network: "flat-network"
platforms:
  peering:
    enabled: true
profile: ambient
tag: ${ISTIO_IMAGE}
terminationGracePeriodSeconds: 29
variant: distroless
EOF
    7. Label the istio-system namespace with the topology.istio.io/network=flat-network label. Note that for Istio to presume a flat network, the network name in this label must match the network name that you set in the spec.values.global.network field.
      kubectl --context=${context1} label namespace istio-system topology.istio.io/network=flat-network --overwrite
kubectl --context=${context2} label namespace istio-system topology.istio.io/network=flat-network --overwrite
    8. Verify that the Istio control plane components are up and running.
      kubectl get pods -n istio-system --context ${context1}     
kubectl get pods -n istio-system --context ${context2}
      Example output:
      NAME                      READY   STATUS    RESTARTS   AGE
istio-cni-node-v8x2f      1/1     Running   0          1m
istiod-54c79986dd-g9lxq   1/1     Running   0          1m
ztunnel-6gccd             1/1     Running   0          50s
NAME                      READY   STATUS    RESTARTS   AGE
istio-cni-node-v5sbf      1/1     Running   0          1m
istiod-76cf85c5d5-z8cjr   1/1     Running   0          1m
ztunnel-wc9w9             1/1     Running   0          50s
    Step 3: Link clusters

    1. Create an east-west gateway in the istio-eastwest namespace. In each cluster, the east-west gateway is implemented as a ztunnel and exposes its xDS server for remote clusters to connect to. This setup facilitates traffic between services across clusters in your multicluster mesh.
      for context in ${context1} ${context2}; do
  kubectl create namespace istio-eastwest --context $context
  istioctl --context=$context multicluster expose --namespace istio-eastwest
done
    2. Verify that the east-west gateways have a PROGRAMMED status of True.
      kubectl get gateway -n istio-eastwest --context ${context1}
kubectl get gateway -n istio-eastwest --context ${context2}
    3. Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters. Note that you can either link the clusters bi-directionally or asymmetrically. In a standard bi-directional setup, services in any of the linked clusters can send requests to and receive requests from the services in any of the other linked clusters. In an asymmetrical setup, you allow one cluster to send requests to another cluster, but the other cluster cannot send requests back to the first cluster.
      In the Solo distribution of Istio 1.29 or later, you can use peering Helm chart to link your clusters.
      1. Get the addresses of the east-west gateway in each cluster. The following commands show examples for two clusters.
        export CLUSTER1_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context1} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")

echo "Cluster1 east-west gateway: $CLUSTER1_EW_ADDRESS"
echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"
      2. Link the clusters by creating Helm releases in each cluster to represent the other clusters. In each cluster where you create Helm releases, a Gateway resource is created that uses the istio-remote GatewayClass. This class allows the gateway to connect to other clusters by using the addresses of the east-west gateways. The following example depicts a bi-directional setup. In an asymmetrical setup, you create Helm releases to only represent the directionality you want to allow.
        helm upgrade -i peering-remote oci://${HELM_REPO}/peering \
  --version ${ISTIO_IMAGE} \
  --namespace istio-eastwest \
  --kube-context ${context1} \
  -f - <<EOF
remote:
  create: true
  items:
    # Remote peer configuration for cluster2
    - name: istio-remote-peer-${cluster2}
      cluster: ${cluster2}
      network: ${cluster2}
      addressType: IPAddress
      address: ${CLUSTER2_EW_ADDRESS}
      trustDomain: cluster.local
      serviceType: ClusterIP
EOF
helm upgrade -i peering-remote oci://${HELM_REPO}/peering \
  --version ${ISTIO_IMAGE} \
  --namespace istio-eastwest \
  --kube-context ${context2} \
  -f - <<EOF
remote:
  create: true
  items:
    # Remote peer configuration for cluster1
    - name: istio-remote-peer-${cluster1}
      cluster: ${cluster1}
      network: ${cluster1}
      addressType: IPAddress
      address: ${CLUSTER1_EW_ADDRESS}
      trustDomain: cluster.local
      serviceType: ClusterIP
EOF
      Use the istioctl multicluster link command to quickly link clusters.
      1. Verify that the contexts for the clusters that you want to include in the multicluster mesh are listed in your kubeconfig file, which is required for the istioctl multicluster link command. If you do not have access to the kubeconfig files, use the declarative resources tabs.
        kubectl config get-contexts
        • In the output, note the names of the cluster contexts, which you use in the next step to link the clusters.
        • If you have multiple kubeconfig files, you can generate a merged kubeconfig file by running the following command.
          KUBECONFIG=<kubeconfig_file1>.yaml:<file2>.yaml kubectl config view --flatten
      2. Using the names of the cluster contexts, link the clusters so that they can communicate. To take a look at the Gateway resources that this command creates, you can include the --generate flag in the command. For more information about this command, see the CLI reference.
        • Bi-directional: You can use the following istioctl command to quickly link the clusters bi-directionally. In each cluster, Gateway resources are created that use the istio-remote GatewayClass. This class allows the gateways to connect to other clusters by using the addresses of the east-west gateways.
          istioctl multicluster link \
  --contexts=$context1,$context2 \
  -n istio-eastwest
          Example output for two clusters:
          Gateway istio-eastwest/istio-remote-peer-cluster1 applied to cluster "<cluster2_context>" pointing to cluster "<cluster1_context>" (network "cluster1")
Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "<cluster1_context>" pointing to cluster "<cluster2_context>" (network "cluster2")
        • Asymmetrical: You can use the following istioctl command to quickly link the clusters asymmetrically. The services in the cluster in the --from flag can send requests to services in the cluster in the --to flag, but sending requests in the reverse direction is not permitted.
          istioctl multicluster link \
  --from ${context1} \
  --to ${context2} \
  -n istio-eastwest
          Example output:
          Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "<cluster1_context>" pointing to cluster "<cluster2_context>" (network "cluster2")
      Link the clusters by declaratively creating istio-remote peer gateways.
      Bi-directional: Use the following Gateway resources to create an istio-remote peer gateway in each cluster. The istio-remote GatewayClass allows the gateways to connect to other clusters by using the addresses of the east-west gateways.
      1. Get the addresses of the east-west gateway in each cluster. The following commands show examples for two clusters.
        export CLUSTER1_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context1} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")

echo "Cluster-1 east-west gateway: $CLUSTER1_EW_ADDRESS"
echo "Cluster-2 east-west gateway: $CLUSTER2_EW_ADDRESS"
      2. Using the east-west gateway addresses, create a Gateway resource in each cluster to represent the other cluster.
        kubectl apply --context ${context1} -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gateway.istio.io/service-account: istio-eastwest
    gateway.istio.io/trust-domain: cluster.local
  labels:
    topology.istio.io/cluster: ${cluster2}
    topology.istio.io/network: flat-network
  name: istio-remote-peer-${cluster2}
  namespace: istio-eastwest
spec:
  addresses:
  - type: IPAddress
    value: $CLUSTER2_EW_ADDRESS
  gatewayClassName: istio-remote
  listeners:
  - name: cross-network
    port: 15008
    protocol: HBONE
    tls:
      mode: Passthrough
    allowedRoutes:
      namespaces: 
        from: Same
  - name: xds-tls
    port: 15012
    protocol: TLS
    tls:
      mode: Passthrough
    allowedRoutes:
      namespaces: 
        from: Same
EOF
kubectl apply --context ${context2} -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gateway.istio.io/service-account: istio-eastwest
    gateway.istio.io/trust-domain: cluster.local
  labels:
    topology.istio.io/cluster: ${cluster1}
    topology.istio.io/network: flat-network
  name: istio-remote-peer-$${cluster1}
  namespace: istio-eastwest
spec:
  addresses:
  - type: IPAddress
    value: $CLUSTER1_EW_ADDRESS
  gatewayClassName: istio-remote
  listeners:
  - name: cross-network
    port: 15008
    protocol: HBONE
    tls:
      mode: Passthrough
    allowedRoutes:
      namespaces: 
        from: Same
  - name: xds-tls
    port: 15012
    protocol: TLS
    tls:
      mode: Passthrough
    allowedRoutes:
      namespaces: 
        from: Same
EOF
      Asymmetrical: You can use the following Gateway resources to create an istio-remote peer gateway in only some clusters. The istio-remote GatewayClass allows the gateway in one cluster to connect to another cluster by using the address of the east-west gateway, but sending requests in the reverse direction is not permitted.
      1. Get the address of the east-west gateway in the cluster that you want to send traffic to. The following command shows an example for cluster-2.
        export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")

echo "Cluster-2 east-west gateway: $CLUSTER2_EW_ADDRESS"
      2. Using the east-west gateway address, create a Gateway resource in the cluster that you want to send requests from. For example, this Gateway resource allows services in cluster-1’s mesh to send requests to services in cluster-2’s mesh through cluster-2’s east-west gateway. However, the reverse is not permitted: services in cluster-2’s mesh cannot send requests through cluster-1’s east-west gateway to services in cluster-1.
        kubectl apply --context ${context1} -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gateway.istio.io/service-account: istio-eastwest
    gateway.istio.io/trust-domain: cluster.local
  labels:
    topology.istio.io/cluster: ${cluster2}
    topology.istio.io/network: flat-network
  name: istio-remote-peer-${cluster2}
  namespace: istio-eastwest
spec:
  addresses:
  - type: IPAddress
    value: $CLUSTER2_EW_ADDRESS
  gatewayClassName: istio-remote
  listeners:
  - name: cross-network
    port: 15008
    protocol: HBONE
    tls:
      mode: Passthrough
    allowedRoutes:
      namespaces: 
        from: Same
  - name: xds-tls
    port: 15012
    protocol: TLS
    tls:
      mode: Passthrough
    allowedRoutes:
      namespaces: 
        from: Same
EOF
    4. Verify that peer linking between your clusters was successful. For more information about this command, see the CLI reference. If any checks fail, run the command with --verbose, and see Validate your multicluster setup.
      istioctl multicluster check --contexts="${context1},${context2}"
      Example output:
      === Cluster: cluster1 ===
✅ Incompatible Environment Variable Check: all relevant environment variables are valid
✅ License Check: license is valid for multicluster
✅ CNI DNS Capture Check: AMBIENT_DNS_CAPTURE is enabled
✅ Pod Check (istiod): all pods healthy
✅ Pod Check (ztunnel): all pods healthy
✅ Pod Check (eastwest gateway istio-eastwest/istio-eastwest): all pods healthy
✅ Gateway Check: all eastwest gateways programmed
     ✅ istio-eastwest/istio-eastwest available at aab8471c7fcfa4a3c82f2d217b015d97-396238517.us-east-1.elb.amazonaws.com
✅ Peers Check: all clusters connected
     ✅ Connected to gloo-gateway-docs-mgt via ab46aa29a49914da789a6d5422aca279-541415195.us-east-2.elb.amazonaws.com
ℹ️  Shared Services Check: no globally shared services found
====== 

=== Cluster: cluster2 ===
✅ Incompatible Environment Variable Check: all relevant environment variables are valid
✅ License Check: license is valid for multicluster
✅ CNI DNS Capture Check: AMBIENT_DNS_CAPTURE is enabled
✅ Pod Check (istiod): all pods healthy
✅ Pod Check (ztunnel): all pods healthy
✅ Pod Check (eastwest gateway istio-eastwest/istio-eastwest): all pods healthy
✅ Gateway Check: all eastwest gateways programmed
     ✅ istio-eastwest/istio-eastwest available at ab46aa29a49914da789a6d5422aca279-541415195.us-east-2.elb.amazonaws.com
✅ Peers Check: all clusters connected
     ✅ Connected to gloo-mesh-core-docs-mgt via aab8471c7fcfa4a3c82f2d217b015d97-396238517.us-east-1.elb.amazonaws.com
ℹ️  Shared Services Check: no globally shared services found
====== 

✅ Intermediate Certs Compatibility Check: all clusters have compatible intermediate certificates
✅ Network Configuration Check: all network configurations are valid
✅ Stale Workloads Check: skipped (flat network not detected)
    5. Optional: Verify that the istiod control plane for each peered cluster is included in each cluster’s proxy status list.
      istioctl proxy-status --context ${context1}
istioctl proxy-status --context ${context2}
      Example output for cluster-1, in which you can verify that the istiod control plane for cluster-2 is listed:
      NAME                                               CLUSTER          ISTIOD                      VERSION                     SUBSCRIBED TYPES
istio-eastwest-67fd5679dc-fhsxs.istio-eastwest     cluster-1        istiod-7b7c9cc4c6-bdm9c     1.28.1-patch0-solo-fips     2 (WADS,WDS)
istiod-6bc6765484-5bbhd.istio-system               cluster-2        istiod-7b7c9cc4c6-bdm9c     1.28.1-patch0-solo-fips     3 (FSDS,SGDS,WDS)
ztunnel-5f8rb.kube-system                          cluster-1        istiod-7b7c9cc4c6-bdm9c     1.28.1-patch0-solo-fips     2 (WADS,WDS)
ztunnel-f96kh.kube-system                          cluster-1        istiod-7b7c9cc4c6-bdm9c     1.28.1-patch0-solo-fips     2 (WADS,WDS)
ztunnel-vtj4f.kube-system                          cluster-1        istiod-7b7c9cc4c6-bdm9c     1.28.1-patch0-solo-fips     2 (WADS,WDS)
    Great job! You have successfully peered multiple clusters in a flat network setup.
    Step 4: Test connectivity

    1. Create the demo namespace and add it to the ambient mesh. Then, deploy the sleep sample app into it. You use this app as a client to test connectivity to the services in the mesh later.
      for context in ${context1} ${context2}; do
  kubectl --context $context create namespace demo
  kubectl --context $context label namespace demo istio.io/dataplane-mode=ambient
  kubectl --context $context apply -n demo -f https://raw.githubusercontent.com/solo-io/doc-examples/main/istio/flat-network/sleep-client.yaml
done
    2. Deploy the global service app. The app is configured to print out Hello version: v1 in cluster-1 and Hello version: v2 in cluster-2. The service has the label solo.io/service-scope: global, which exposes the app under a common domain name global-service.demo.mesh.internal across both of your clusters. For more information, see Make services available across clusters.
      curl -L https://raw.githubusercontent.com/solo-io/doc-examples/main/istio/flat-network/global-service.yaml -o global-service.yaml
sed 's/VERSION_PLACEHOLDER/v1/g' global-service.yaml | kubectl --context ${context1} apply -n demo -f -
sed 's/VERSION_PLACEHOLDER/v2/g' global-service.yaml | kubectl --context ${context2} apply -n demo -f -
    3. Verify that you see the following Istio resources in your cluster.
      • An Istio ServiceEntry for the global service name.
      • An Istio WorkloadEntry with the IP address for each global service pod that runs in the opposite cluster. For example, cluster-1 has two WorkloadEntries. Each WorkloadEntry points to the IP address of one global service instance in cluster-2. The WorkloadEntry resources are used to route traffic to the pod IP addresses directly without the need for the east-west gateway.
      for context in ${context1} ${context2}; do
  kubectl get serviceentry -n istio-system --context $context
  kubectl get workloadentry -n istio-system --context $context
done
      Example output:
      NAME                          HOSTS                                   LOCATION   RESOLUTION   AGE
autogen.demo.global-service   ["global-service.demo.mesh.internal"]              STATIC       2m25s
NAME                                                                     AGE     ADDRESS
autogen.cluster-2.demo.global-service                                    2m26s   
autogenflat.cluster-2.demo.global-service-d955b94d5-jfzkt.eacd81ea03fd   2m25s   10.20.0.13
autogenflat.cluster-2.demo.global-service-d955b94d5-tgmcn.eacd81ea03fd   2m25s   10.20.0.12

NAME                          HOSTS                                   LOCATION   RESOLUTION   AGE
autogen.demo.global-service   ["global-service.demo.mesh.internal"]              STATIC       2m26s
NAME                                                                      AGE     ADDRESS
autogen.cluster-1.demo.global-service                                     2m26s   
autogenflat.cluster-1.demo.global-service-5d86865969-552td.eacd81ea03fd   2m25s   10.10.0.17
autogenflat.cluster-1.demo.global-service-5d86865969-tp5hr.eacd81ea03fd   2m25s   10.10.0.18
    4. Send a curl request from the example sleep app in cluster-1 to the global service name. In your CLI output, verify that you see replies from the global service app from both of your clusters.
      kubectl --context ${context1} exec -n demo deploy/sleep -- sh -c "
for i in \$(seq 1 10); do
  curl -s global-service.demo.mesh.internal:5000/hello
  echo
done" | grep -o 'version: v[0-9]' | sort | uniq -c
      Example output:
      5 version: v1
5 version: v2
      Note that you might see a different CLI output, such as 6 version: v1 4 version: v2. The more requests you send, the closer you get to a 50:50 distribution of requests.
    5. Get the name of the ztunnel instance in cluster-1.
      kubectl get pods -n istio-system --context ${context1} | grep ztunnel
    6. Review the ztunnel logs. In your output, verify that you see log entries from the sleep app in cluster-1 to one of the global service IP addresses in cluster-2. In this example, the sleep app in cluster-1 has an IP address of 10.10.0.14 and reaches out to the global service instance with the 10.20.0.12 IP address in cluster-2.
      kubectl logs <ztunnel-pod> -n istio-system --context ${context1}
      Example output:
      2025-09-11T19:42:33.699609Z	info	access	connection complete	src.addr=10.10.0.14:40292 src.
workload="sleep-746f9d766c-pmk5f" src.namespace="demo" src.identity="spiffe://cluster.local/ns/demo/sa/
sleep" dst.addr=10.20.0.12:15008 dst.hbone_addr=10.20.0.12:5000 dst.service="global-service.demo.mesh.
internal" dst.workload="autogenflat.cluster-2.demo.global-service-d955b94d5-7bkck.eacd81ea03fd" dst.
namespace="demo" dst.identity="spiffe://cluster.local/ns/demo/sa/default" direction="outbound" 
bytes_sent=107 bytes_recv=218 duration="40ms"
    Next

    You can now follow other guides to expand your ambient mesh capabilities.
    Cleanup

    You can optionally remove the kind cluster setup in this guide.
    1. Remove the kind clusters.
      kind delete cluster --name cluster-1
kind delete cluster --name cluster-2
    2. Remove the cloud-provider-kind CLI from your machine. The CLI is typically installed into your $GOPATH.
      echo $GOPATH
rm $GOPATH/go/bin/cloud-provider-kind
    Optional: Validate your multicluster setup

    Both before and after you link clusters into a multicluster mesh, you can use the istioctl multicluster check command, along with other observability checks, to verify multiple aspects of multicluster ambient mesh support and status.
    istioctl multicluster check

    You can use the istioctl multicluster check --precheck command to check the individual readiness of each cluster before running istioctl multicluster link to link them in a multicluster mesh, and run it again after linking to confirm that the connections were successful. This command performs checks listed in the following sections, which you can review to understand what each check validates. Additionally, if any of the checks fail, run the command with the --verbose option, and review the following troubleshooting recommendations.
    istioctl multicluster check --verbose --contexts="$context1,$context2"
    For more information about this command, see the CLI reference.
    Incompatible environment variables
    Checks whether the ENABLE_PEERING_DISCOVERY=true and optionally K8S_SELECT_WORKLOAD_ENTRIES=true environment variables are set incorrectly or are not supported for multicluster ambient mesh.
    Example verbose output:
    --- Incompatible Environment Variable Check ---

✅ Incompatible Environment Variable Check: K8S_SELECT_WORKLOAD_ENTRIES is valid ("")
✅ Incompatible Environment Variable Check: ENABLE_PEERING_DISCOVERY is valid ("true")
✅ Incompatible Environment Variable Check: all relevant environment variables are valid
    If this check fails, check your environment variables in your istiod configuration, such as by running helm get values --kube-context ${CLUSTER_CONTEXT} istiod -n istio-system -o yaml, and update your configuration.
    License validity
    Checks whether the license in use by istiod is valid for multicluster ambient mesh. Multicluster capabilities require an Enterprise level license for Solo Enterprise for Istio.
    Example verbose output:
    --- License Check ---

✅ License Check: license is valid for multicluster
    If your license does not support multicluster ambient mesh, contact your Solo account representative.
    Pod health
    Checks the health of the pods in the cluster. All istiod, ztunnel, and east-west gateway pods across the checked clusters must be healthy and running for the multicluster mesh to function correctly.
    Example verbose output:
    --- Pod Check (istiod) ---

NAME                        READY     STATUS      RESTARTS     AGE
istiod-6d9cdf88cf-l47tf     1/1       Running     0            10m18s

✅ Pod Check (istiod): all pods healthy


--- Pod Check (ztunnel) ---

NAME              READY     STATUS      RESTARTS     AGE
ztunnel-dvlwk     1/1       Running     0            10m6s

✅ Pod Check (ztunnel): all pods healthy


--- Pod Check (eastwest gateway) ---

NAME                                READY     STATUS      RESTARTS     AGE
istio-eastwest-857b77fc5d-qgnrl     1/1       Running     0            9m33s

✅ Pod Check (eastwest gateway): all pods healthy
    To check any unhealthy pods, run the following commands. Consider checking the pod logs, and review Debug Istio.
    kubectl get po -n istio-system
kubectl get po -n istio-eastwest
    East-west gateway status
    Checks the status of the east-west gateways in the cluster. When an east-west gateway is created, the gateway controller creates a Kubernetes service to expose the gateway. Once this service is correctly attached to the gateway and has an address assigned, the east-west gateway has a Programmed status of true.
    Example verbose output:
    --- Gateway Check ---

Gateway: istio-eastwest
Addresses:
- 172.18.7.110
Status: programmed ✅

✅ Gateway Check: all eastwest gateways programmed
    If the Programmed status is not true, an issue might exist with the address allocation for the service. Check the east-west gateway with a command such as kubectl get svc -n istio-eastwest, and verify that your cloud provider can correctly allocate addresses to the service.
    Remote peer gateway status
    Checks the status of the remote peer gateways in the cluster, which represent the other peered clusters in the multicluster setup. These remote gateways configure the connection between the local cluster’s istiod control plane, and the peered clusters’ remote networks to enable xDS communication between peers. When the initial network connection between istiod and a remote peer is made, the gateway’s gloo.solo.io/PeerConnected status updates to true. Then, when the full xDS sync occurs between peers, the gateway’s gloo.solo.io/PeeringSucceeded status also updates to true. This check ensures that both statuses are true.
    Example verbose output:
    --- Peers Check ---

Cluster: cluster2
Addresses:
- 172.18.7.130
Conditions:
- Accepted: True
- Programmed: True
- gloo.solo.io/PeerConnected: True
- gloo.solo.io/PeeringSucceeded: True
- gloo.solo.io/PeerDataPlaneProgrammed: True
Status: connected ✅

✅ Peers Check: all clusters connected
    If the connection is severed between the peers, the gloo.solo.io/PeerConnected status becomes false. A failed connection between peers can be due to either a misconfiguration in the peering setup, or a network issue blocking port 15008 on the remote cluster, which is the cross-network HBONE port that the east-west gateway listens on. Review the steps you took to link clusters together, such as the steps outlined in the Helm default network guide. Additionally, review any firewall rules or network policies that might block access through port 15008 on the remote cluster.
    Intermediate certificate compatibility
    Confirms the certificate compatibility between peered clusters. This check reads the root-cert.pem from the istio-ca-root-cert configmap in the istio-system namespace, and uses x509 certificate validation to confirm the root cert is compatible with all of the clusters’ ca-cert.pem intermediate certificate chains from the cacerts secret.
    Example verbose output:
    --- Intermediate Certs Compatibility Check ---

ℹ  Intermediate Certs Compatibility Check: cluster cluster1 root certificate SHA256 sum: 6d18f32e134824c158d97f32618657c45d5a83839f838ada751757139481537e
ℹ  Intermediate Certs Compatibility Check: cluster cluster2 root certificate SHA256 sum: 6d18f32e134824c158d97f32618657c45d5a83839f838ada751757139481537e
✅ Intermediate Certs Compatibility Check: cluster cluster1 has compatible intermediate certificates with cluster cluster2 
✅ Intermediate Certs Compatibility Check: cluster cluster2 has compatible intermediate certificates with cluster cluster1 
✅ Intermediate Certs Compatibility Check: all clusters have compatible intermediate certificates
    If this check fails because the root certs are not valid for each peered clusters’ intermediate certificate chain, you can check the istiod logs for TLS errors when attempting to communicate with a peered cluster, such as the following:
    2025-12-04T22:09:22.474517Z     warn    deltaadsc       disconnected, retrying in 24.735483751s: delta stream: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: remote error: tls: unknown certificate authority"       target=peering-cluster2
    Ensure each cluster has a cacerts secret in the istio-system namespace. To regenerate invalid certificates for each cluster, follow the example steps in Create a shared root of trust.
    Network configuration
    Confirms the network configuration of the multicluster mesh. For multicluster peering setups that do not use a flat network topology, each cluster must occupy a unique network. The network name must be defined with the label topology.istio.io/network and set on both the istio-system namespace and the istio-eastwest gateway resource. The same network name must also be set as the NETWORK environment variable on the ztunnel daemonset. Each remote gateway that represents that cluster must have the topology.istio.io/network label equal to the network of the remote cluster.
    Example verbose output:
    --- Network Configuration Check ---

✅ Cluster cluster1 has network: cluster1
✅ Eastwest gateway istio-eastwest/istio-eastwest has correct network label: cluster1
✅ Cluster cluster2 has network: cluster2
✅ Eastwest gateway istio-eastwest/istio-eastwest has correct network label: cluster2
✅ Remote gateway istio-eastwest/istio-remote-peer-cluster2 references network cluster2 (clusters: [cluster2])
✅ Remote gateway istio-eastwest/istio-remote-peer-cluster1 references network cluster1 (clusters: [cluster1])
✅ Network Configuration Check: all network configurations are valid
    Mismatched network identities cause errors in cross-cluster communication, which leads to error logs in ztunnel pods that indicate a network timeout on the outbound communication. Notably, the destination address on these errors is a 240.X.X.X address, instead of the correct remote peer gateway address. You can run kubectl logs -l app=ztunnel -n istio-system --tail=10 --context ${CLUSTER_CONTEXT} | grep -iE "error|warn" to review logs such as the following:
    2025-11-18T16:14:53.490573Z     error   access  connection complete     src.addr=240.0.2.27:46802 src.workload="ratings-v1-5dc79b6bcd-zm8v6" src.namespace="bookinfo" src.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings" dst.addr=240.0.9.43:15008 dst.hbone_addr=240.0.9.43:9080 dst.service="productpage.bookinfo.mesh.internal" dst.workload="autogenflat.portfolio1-soloiopoc-cluster1.bookinfo.productpage-v1-54bb874995-hblwp.ee508601917c" dst.namespace="bookinfo" dst.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-productpage" direction="outbound" bytes_sent=0 bytes_recv=0 duration="10001ms" error="connection timed out, maybe a NetworkPolicy is blocking HBONE port 15008: deadline has elapsed"
    To troubleshoot these issues, be sure that you use unique network names to represent each cluster, and that you correctly labeled the cluster’s istio-system namespace with that network name, such as by running kubectl label namespace istio-system --context ${CLUSTER_CONTEXT} topology.istio.io/network=${CLUSTER_NAME}. You can also relabel the east-west gateway in the cluster, and the remote peer gateways in other clusters that represent this cluster.
    Stale workload entries
    In flat network setups, checks for any outdated workload entries that must be removed from the multicluster mesh. Stale workload entries might exist from pods that were deleted, but the autogenerated entries for those workloads were not correctly cleaned up. If you do not use a flat network topology, no autogenerated workload entries exist to be validated, and this check can be ignored.
    Example verbose output for a non-flat network setup:
    --- Stale Workloads Check ---

⚠  Stale Workloads Check: no autogenflat workload entries found
    If you use a flat network topology, and this check fails with stale workload entries, run kubectl get workloadentries -n istio-system | grep autogenflat to list the autogenerated workload entries in the remote cluster, and compare the list to the output of kubectl get pods in the source cluster for those workloads. You can safely manually delete the stale workload entries in the remote cluster for pods that no longer exist in the source cluster, such as by running kubectl get workloadentries -n istio-system <entry_name>.
    Further debugging and observability

    For additional guidance around observing your multicluster ambient mesh, check out the observability overview, which contains links to guides on using logs, metrics, and traces in your Istio environment.
    For additional guidance around debugging your multicluster ambient mesh, check out the Istio troubleshooting guide.
    Default setup