Create a multicluster mesh with Helm

Considerations link

Before you set up a multicluster ambient mesh, review the following considerations and requirements.

License requirements

Version requirements

In Gloo Mesh version 2.7 and later, multicluster setups require the Solo distribution of Istio version 1.24.3 or later (1.24.3-solo), including the Solo distribution of istioctl.

Revision and canary upgrade limitations

The upgrade guides in this documentation show you how to perform in-place upgrades for your Istio components, which is the recommended upgrade strategy.

Overview link

The following diagram demonstrates an ambient mesh setup across multiple clusters. In this guide, you deploy an ambient mesh to each workload cluster, create an east-west gateway in each cluster, and link the istiod control planes across cluster networks by using peering gateways. In the next guide, you can deploy the Bookinfo sample app to the ambient mesh in each cluster, and make select services available across the multicluster mesh. Incoming requests can then be routed from an ingress gateway, such as Gloo Gateway, to services in your mesh across all clusters. For more information about in-mesh routing, check out Control in-mesh traffic with east-west gateways and waypoints.

For more information about the components that are installed in these steps, see the ambient components overview.

Upgrade and link existing ambient meshes link

Upgrade your existing ambient meshes installed with Helm and link them to create a multicluster ambient mesh.

Set up tools link

1. Set your Enterprise level license for Gloo Mesh as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing.

     export GLOO_MESH_LICENSE_KEY=<enterprise_license_key>
     

2. Save the details for the version of the Solo distribution of Istio that your ambient meshes run. In Gloo Mesh version 2.7 and later, multicluster setups require version 1.24.3 or later. If you must upgrade the istiod versions to 1.24 or later, you can use these environment variables to upgrade each istiod instance in later steps.

     # Change the tags as needed
   export ISTIO_IMAGE=1.25.2-solo
   export REPO_KEY=e038d180f90a
   export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
   export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}
     

   notifications

   In the Solo distribution of Istio 1.25 and later, you can access enterprise-level features by passing your Solo license in the license.value or license.secretRef field of your Solo istiod Helm chart. The Helm chart that is provided by Solo includes safeguards, default settings, and upgrade handling to ensure a reliable and secure Istio deployment. Although you can pass the license key in the pilot.env.SOLO_LICENSE_KEY field of the open source Istio Helm chart, this method is not recommended. For best results, we strongly recommend using the Solo Helm chart to install and manage Istio. For example, to install the Solo distribution of 1.25, you can run the helm install istiod oci://us-docker.pkg.dev/gloo-mesh/istio-helm-e038d180f90a/istiod ... command and provide your installation values in --set flags or a -f <file.yaml> config file.

   1. Save the Solo distribution of Istio patch version and tag.

        # Change the tags as needed
      export ISTIO_IMAGE=1.24.5-solo
        

   2. Save the repo key for version 1.24 of the Solo distribution of Istio. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Ambient section of the Istio images built by Solo.io support article.

        # 12-character hash at the end of the 1.24 repo URL
      export REPO_KEY=<repo_key>
      export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
      export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}
        

3. Get the Solo distribution of Istio binary and install istioctl, which you use for multicluster linking and gateway commands.

   1. Get the OS and architecture that you use on your machine.

        OS=$(uname | tr '[:upper:]' '[:lower:]' | sed -E 's/darwin/osx/')
      ARCH=$(uname -m | sed -E 's/aarch/arm/; s/x86_64/amd64/; s/armv7l/armv7/')
      echo $OS
      echo $ARCH
        

   2. Download the Solo distribution of Istio binary and install istioctl.

        mkdir -p ~/.istioctl/bin
      curl -sSL https://storage.googleapis.com/istio-binaries-$REPO_KEY/$ISTIO_IMAGE/istioctl-$ISTIO_IMAGE-$OS-$ARCH.tar.gz | tar xzf - -C ~/.istioctl/bin
      chmod +x ~/.istioctl/bin/istioctl
      
      export PATH=${HOME}/.istioctl/bin:${PATH}
        

   3. Verify that the istioctl client runs the Solo distribution of Istio that your ambient meshes run.

        istioctl version --remote=false
        

      Example output:

        client version: 1.25.2-solo
        

Create a shared root of trust link

Each cluster in the multicluster setup must have a shared root of trust. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.

  curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}

mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

function create_cacerts_secret() {
  context=${1:?context}
  cluster=${2:?cluster}
  make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
  kubectl --context=${context} create ns istio-system || true
  kubectl --context=${context} create secret generic cacerts -n istio-system \
    --from-file=${cluster}/ca-cert.pem \
    --from-file=${cluster}/ca-key.pem \
    --from-file=${cluster}/root-cert.pem \
    --from-file=${cluster}/cert-chain.pem
}

create_cacerts_secret ${REMOTE_CONTEXT1} ${REMOTE_CLUSTER1}
create_cacerts_secret ${REMOTE_CONTEXT2} ${REMOTE_CLUSTER2}

cd ../..
  

Upgrade settings link

In each cluster, update the ambient mesh components for multicluster, and create an east-west gateway so that traffic requests can be routed cross-cluster.

1. Save the name and kubeconfig context of a cluster where you run an ambient mesh. Each time you repeat the steps in this guide, you change these variables to the next workload cluster’s name and context.

     export CLUSTER_NAME=<cluster-name>
   export CLUSTER_CONTEXT=<cluster-context>
     

2. Get the current values for the istiod Helm release in your cluster.

     helm get values --kube-context ${CLUSTER_CONTEXT} istiod -n istio-system -o yaml > istiod.yaml
     

3. Update your Helm release with the following multicluster values. If you must update the Istio minor version, include the --set global.tag=${ISTIO_IMAGE} and --set global.hub=${REPO} flags too.

   • Istio 1.25 and later: If you prefer to specify your license secret instead of an inline value, you can use the --set license.secretRef.name and --set license.secretRef.namespace flags instead.

       helm upgrade istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f istiod.yaml \
     --set env.PILOT_ENABLE_IP_AUTOALLOCATE="true" \
     --set env.PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES="false" \
     --set env.PILOT_SKIP_VALIDATE_TRUST_DOMAIN="true" \
     --set global.multiCluster.clusterName=${CLUSTER_NAME} \
     --set global.network=${CLUSTER_NAME} \
     --set global.trustDomain="${CLUSTER_NAME}.local" \
     --set platforms.peering.enabled=true \
     --set license.value=${GLOO_MESH_LICENSE_KEY}
       

   • Istio 1.24 and earlier:

       helm upgrade istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f istiod.yaml \
     --set env.PILOT_ENABLE_IP_AUTOALLOCATE="true" \
     --set env.PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES="false" \
     --set env.PILOT_SKIP_VALIDATE_TRUST_DOMAIN="true" \
     --set global.multiCluster.clusterName=${CLUSTER_NAME} \
     --set global.network=${CLUSTER_NAME} \
     --set global.trustDomain="${CLUSTER_NAME}.local" \
     --set platforms.peering.enabled=true
       

   • Istio 1.25 and later: If you prefer to specify your license secret instead of an inline value, you can use the --set license.secretRef.name and --set license.secretRef.namespace flags instead.

       helm upgrade istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f istiod.yaml \
     --set env.PILOT_ENABLE_IP_AUTOALLOCATE="true" \
     --set env.PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES="false" \
     --set env.PILOT_SKIP_VALIDATE_TRUST_DOMAIN="true" \
     --set global.multiCluster.clusterName=${CLUSTER_NAME} \
     --set global.network=${CLUSTER_NAME} \
     --set global.platform=openshift \
     --set global.trustDomain="${CLUSTER_NAME}.local" \
     --set platforms.peering.enabled=true \
     --set license.value=${GLOO_MESH_LICENSE_KEY}
       

   • Istio 1.24 and earlier:

       helm upgrade istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f istiod.yaml \
     --set env.PILOT_ENABLE_IP_AUTOALLOCATE="true" \
     --set env.PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES="false" \
     --set env.PILOT_SKIP_VALIDATE_TRUST_DOMAIN="true" \
     --set global.multiCluster.clusterName=${CLUSTER_NAME} \
     --set global.network=${CLUSTER_NAME} \
     --set global.platform=openshift \
     --set global.trustDomain="${CLUSTER_NAME}.local" \
     --set platforms.peering.enabled=true
       

4. Verify that the istiod pods are successfully restarted. Note that it might take a few seconds for the pods to become available.

     kubectl get pods --context ${CLUSTER_CONTEXT} -n istio-system | grep istiod
     

   Example output:

     istiod-b84c55cff-tllfr   1/1     Running   0          58s
     

5. Get the current values for the ztunnel Helm release in your cluster.

     helm get values ztunnel --kube-context ${CLUSTER_CONTEXT} -n istio-system -o yaml > ztunnel.yaml
     

     helm get values ztunnel --kube-context ${CLUSTER_CONTEXT} -n kube-system -o yaml > ztunnel.yaml
     

6. Update your Helm release with the following multicluster values. If you must update the Istio minor version, include the --set tag=${ISTIO_IMAGE} and --set hub=${REPO} flags too.

     helm upgrade ztunnel oci://${HELM_REPO}/ztunnel \
   -n istio-system \
   --version ${ISTIO_IMAGE} \
   --kube-context ${CLUSTER_CONTEXT} \
   -f ztunnel.yaml \
   --set env.SKIP_VALIDATE_TRUST_DOMAIN="true" \
   --set multiCluster.clusterName=${CLUSTER_NAME} \
   --set network=${CLUSTER_NAME}
     

     helm upgrade ztunnel oci://${HELM_REPO}/ztunnel \
   -n kube-system \
   --version ${ISTIO_IMAGE} \
   --kube-context ${CLUSTER_CONTEXT} \
   -f ztunnel.yaml \
   --set env.SKIP_VALIDATE_TRUST_DOMAIN="true" \
   --set multiCluster.clusterName=${CLUSTER_NAME} \
   --set network=${CLUSTER_NAME}
     

7. Verify that the ztunnel pods are successfully installed. Because the ztunnel is deployed as a daemon set, the number of pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.

     kubectl get pods -A --context ${CLUSTER_CONTEXT} | grep ztunnel
     

   Example output:

     ztunnel-tvtzn             1/1     Running   0          7s
   ztunnel-vtpjm             1/1     Running   0          4s
   ztunnel-hllxg             1/1     Running   0          4s
     

8. Label the istio-system namespace with the cluster’s network name, which you previously set to your cluster’s name in the global.network field of the istiod installation. The ambient control plane uses this label internally to group pods that exist in the same L3 network.

     kubectl label namespace istio-system --context ${CLUSTER_CONTEXT} topology.istio.io/network=${CLUSTER_NAME}
     

9. Create an east-west gateway in the istio-eastwest namespace. An east-west gateway facilitates traffic between services in each cluster in your multicluster mesh.

   • You can use the following istioctl command to quickly create the east-west gateway.

       kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
     istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT}
       

   • To take a look at the Gateway resource that this command creates, you can include the --generate flag in the command.

       kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
     istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT} --generate
       

     In this example output, the gatewayClassName that is used, istio-eastwest, is included by default when you install Istio in ambient mode.

       apiVersion: gateway.networking.k8s.io/v1
     kind: Gateway
     metadata:
       labels:
         istio.io/expose-istiod: "15012"
         topology.istio.io/network: "<cluster_network_name>"
       name: istio-eastwest
       namespace: istio-eastwest
     spec:
       gatewayClassName: istio-eastwest
       listeners:
       - name: cross-network
         port: 15008
         protocol: HBONE
         tls:
           mode: Passthrough
       - name: xds-tls
         port: 15012
         protocol: TLS
         tls:
           mode: Passthrough
       

10. Verify that the east-west gateway is successfully deployed.

      kubectl get pods -n istio-eastwest --context $CLUSTER_CONTEXT
      

11. For each cluster that you want to add to the multicluster ambient mesh setup, repeat these steps to upgrade the Helm values and deploy an east-west gateway. Remember to change the cluster name and context variables each time you repeat the steps.

      export CLUSTER_NAME=<cluster-name>
    export CLUSTER_CONTEXT=<cluster-context>
      

Link clusters link

Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters.

1. Verify that the contexts for the clusters that you want to include in the multicluster mesh are listed in your kubeconfig file.

     kubectl config get-contexts
     

   • In the output, note the names of the cluster contexts, which you use in the next step to link the clusters.
   • If you have multiple kubeconfig files, you can generate a merged kubeconfig file by running the following command.

       KUBECONFIG=<kubeconfig_file1>.yaml:<file2>.yaml:<file3>.yaml kubectl config view --flatten
       

2. Using the names of the cluster contexts, link the clusters so that they can communicate. Note that you can either link the clusters bi-directionally or asymmetrically. In a standard bi-directional setup, services in any of the linked clusters can send requests to and receive requests from the services in any of the other linked clusters. In an asymmetrical setup, you allow one cluster to send requests to another cluster, but the other cluster cannot send requests back to the first cluster.

   You can use the following istioctl command to quickly link the clusters bi-directionally. In each cluster, Gateway resources are created that use the istio-remote GatewayClass. This class allows the gateways to connect to other clusters by using the clusters’ contexts.

     istioctl multicluster link --namespace istio-eastwest --contexts=<context1>,<context2>,<context3>
     

   To take a look at the Gateway resources that this command creates, you can include the --generate flag in the command.

   Example output for two clusters:

     Gateway istio-eastwest/istio-remote-peer-cluster1 applied to cluster "<cluster2_context>" pointing to cluster "<cluster1_context>" (network "cluster1")
   Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "<cluster1_context>" pointing to cluster "<cluster2_context>" (network "cluster2")
     

   You can use the following istioctl command to quickly link the clusters asymmetrically. The services in the cluster in the --from flag can send requests to services in cluster in the --to flag, but sending requests in the reverse direction is not permitted.

     istioctl multicluster link --namespace istio-eastwest --from <context1> --to <context2>
     

   To take a look at the Gateway resources that this command creates, you can include the --generate flag in the command.

   For example, this command allows services in cluster1’s mesh to send requests to services in cluster2’s mesh through cluster2’s east-west gateway. However, the reverse is not permitted: services in cluster2’s mesh cannot send requests through cluster1’s east-west gateway to services in cluster1.

     istioctl multicluster link --namespace istio-eastwest --from cluster1 --to cluster2
     

   Example output:

     Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "<cluster1_context>" pointing to cluster "<cluster2_context>" (network "cluster2")
     

  kubectl get gateways -n istio-eastwest --context $CLUSTER_CONTEXT
  

  NAME                                     CLASS            ADDRESS     PROGRAMMED   AGE
istio-eastwest                           istio-eastwest   <address>   True         29m
istio-remote-peer-cluster1               istio-remote     <address>   True         16m
...
  

  kubectl get svc -n istio-eastwest --context $CLUSTER_CONTEXT
  

  NAME             TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                           AGE
istio-eastwest   LoadBalancer   172.20.44.255   <address>       15021:31730/TCP,15008:30632/TCP,15012:32356/TCP   29m
  

Next: Add apps to the ambient mesh. For multicluster setups, this includes making specific services available across your linked cluster setup.

Install and link new ambient meshes link

In each cluster, use Helm to create the ambient mesh components. Then, create an east-west gateway so that traffic requests can be routed cross-cluster, and link clusters to enable cross-cluster service discovery.

Set up tools link

1. Set your Enterprise level license for Gloo Mesh as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing.

     export GLOO_MESH_LICENSE_KEY=<enterprise_license_key>
     

2. Choose the version of Istio that you want to install or upgrade to by reviewing the supported versions. In Gloo Mesh version 2.7 and later, multicluster setups require version 1.24.3 or later.

3. Save the details for the version of the Solo distribution of Istio that you want to install.

     export ISTIO_VERSION=1.25.2
   export ISTIO_IMAGE=${ISTIO_VERSION}-solo
   export REPO_KEY=e038d180f90a
     

   1. Save the Solo distribution of Istio patch version.

        export ISTIO_VERSION=1.24.5
      export ISTIO_IMAGE=${ISTIO_VERSION}-solo
        

   2. Save the repo key for version 1.24 of the Solo distribution of Istio. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Ambient section of the Istio images built by Solo.io support article.

        # 12-character hash at the end of the 1.24 repo URL
      export REPO_KEY=<repo_key>
        

4. Get the Solo distribution of Istio binary and install istioctl, which you use for multicluster linking and gateway commands.

   1. Get the OS and architecture that you use on your machine.

        OS=$(uname | tr '[:upper:]' '[:lower:]' | sed -E 's/darwin/osx/')
      ARCH=$(uname -m | sed -E 's/aarch/arm/; s/x86_64/amd64/; s/armv7l/armv7/')
      echo $OS
      echo $ARCH
        

   2. Download the Solo distribution of Istio binary and install istioctl.

        mkdir -p ~/.istioctl/bin
      curl -sSL https://storage.googleapis.com/istio-binaries-$REPO_KEY/$ISTIO_IMAGE/istioctl-$ISTIO_IMAGE-$OS-$ARCH.tar.gz | tar xzf - -C ~/.istioctl/bin
      chmod +x ~/.istioctl/bin/istioctl
      
      export PATH=${HOME}/.istioctl/bin:${PATH}
        

   3. Verify that the istioctl client runs the Solo distribution of Istio that you want to install.

        istioctl version --remote=false
        

      Example output:

        client version: 1.25.2-solo
        

Create a shared root of trust link

Each cluster in the multicluster setup must have a shared root of trust. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.

  curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}

mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

function create_cacerts_secret() {
  context=${1:?context}
  cluster=${2:?cluster}
  make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
  kubectl --context=${context} create ns istio-system || true
  kubectl --context=${context} create secret generic cacerts -n istio-system \
    --from-file=${cluster}/ca-cert.pem \
    --from-file=${cluster}/ca-key.pem \
    --from-file=${cluster}/root-cert.pem \
    --from-file=${cluster}/cert-chain.pem
}

create_cacerts_secret ${REMOTE_CONTEXT1} ${REMOTE_CLUSTER1}
create_cacerts_secret ${REMOTE_CONTEXT2} ${REMOTE_CLUSTER2}

cd ../..
  

Deploy ambient components link

1. Save the name and kubeconfig context of a cluster where you want to install Istio in the following environment variables. Each time you repeat the steps in this guide, you change these variables to the next workload cluster’s name and context.

     export CLUSTER_NAME=<cluster-name>
   export CLUSTER_CONTEXT=<cluster-context>
     

2. Apply the CRDs for the Kubernetes Gateway API to your cluster, which are required to create components such as waypoint proxies for L7 traffic policies, gateways with the Gateway resource, and more.

     kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v/standard-install.yaml --context ${CLUSTER_CONTEXT}
     

3. Install the base chart, which contains the CRDs and cluster roles required to set up Istio.

     helm upgrade --install istio-base oci://${HELM_REPO}/base \
   --namespace istio-system \
   --create-namespace \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   defaultRevision: ""
   profile: ambient
   EOF
     

     helm upgrade --install istio-base oci://${HELM_REPO}/base \
   --namespace istio-system \
   --create-namespace \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   defaultRevision: ""
   profile: ambient
   global:
     platform: openshift
   EOF
     

4. Create the istiod control plane in your cluster.

   • Istio 1.25 and later: If you prefer to specify your license secret instead of an inline value, you can uncomment the secretRef fields.

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     env:
       # Assigns IP addresses to multicluster services
       PILOT_ENABLE_IP_AUTOALLOCATE: "true"
       # Disable selecting workload entries for local service routing.
       # Required for Gloo VirtualDestinaton functionality.
       PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
       # Required when meshConfig.trustDomain is set
       PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
     global:
       hub: ${REPO}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
     istio_cni:
       namespace: istio-system
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       # Assign each cluster a unique trust domain to apply policies to specific clusters
       trustDomain: "${CLUSTER_NAME}.local"
     # Required to enable multicluster support
     platforms:
       peering:
         enabled: true
     profile: ambient
     license:
       value: ${GLOO_MESH_LICENSE_KEY}
       # secretRef:
       #   name: 
       #   namespace: 
     EOF
       

   • Istio 1.24 and earlier:

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     env:
       # Assigns IP addresses to multicluster services
       PILOT_ENABLE_IP_AUTOALLOCATE: "true"
       # Disable selecting workload entries for local service routing.
       # Required for Gloo VirtualDestinaton functionality.
       PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
       # Required when meshConfig.trustDomain is set
       PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
     global:
       hub: ${REPO}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
     istio_cni:
       namespace: istio-system
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       # Assign each cluster a unique trust domain to apply policies to specific clusters
       trustDomain: "${CLUSTER_NAME}.local"
     # Required to enable multicluster support
     platforms:
       peering:
         enabled: true
     profile: ambient
     EOF
       

   • Istio 1.25 and later: If you prefer to specify your license secret instead of an inline value, you can uncomment the secretRef fields.

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     env:
       # Assigns IP addresses to multicluster services
       PILOT_ENABLE_IP_AUTOALLOCATE: "true"
       # Disable selecting workload entries for local service routing.
       # Required for Gloo VirtualDestinaton functionality.
       PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
       # Required when meshConfig.trustDomain is set
       PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
     global:
       hub: ${REPO}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
       platform: openshift
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
     istio_cni:
       namespace: kube-system
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       # Assign each cluster a unique trust domain to apply policies to specific clusters
       trustDomain: "${CLUSTER_NAME}.local"
     # Required to enable multicluster support
     platforms:
       peering:
         enabled: true
     profile: ambient
     license:
       value: ${GLOO_MESH_LICENSE_KEY}
       # secretRef:
       #   name: 
       #   namespace: 
     EOF
       

   • Istio 1.24 and earlier:

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     env:
       # Assigns IP addresses to multicluster services
       PILOT_ENABLE_IP_AUTOALLOCATE: "true"
       # Disable selecting workload entries for local service routing.
       # Required for Gloo VirtualDestinaton functionality.
       PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
       # Required when meshConfig.trustDomain is set
       PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
     global:
       hub: ${REPO}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
       platform: openshift
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
     istio_cni:
       namespace: kube-system
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       # Assign each cluster a unique trust domain to apply policies to specific clusters
       trustDomain: "${CLUSTER_NAME}.local"
     # Required to enable multicluster support
     platforms:
       peering:
         enabled: true
     profile: ambient
     EOF
       

5. Install the Istio CNI node agent daemonset. Note that although the CNI is included in this section, it is technically not part of the control plane or data plane.

     helm upgrade --install istio-cni oci://${HELM_REPO}/cni \
   --namespace istio-system \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   # Assigns IP addresses to multicluster services
   ambient:
     dnsCapture: true
   excludeNamespaces:
     - istio-system
     - kube-system
   global:
     hub: ${REPO}
     tag: ${ISTIO_IMAGE}
   profile: ambient
   EOF
     

     helm upgrade --install istio-cni oci://${HELM_REPO}/cni \
   --namespace kube-system \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   # Assigns IP addresses to multicluster services
   ambient:
     dnsCapture: true
   excludeNamespaces:
     - istio-system
     - kube-system
   global:
     hub: ${REPO}
     platform: openshift
     tag: ${ISTIO_IMAGE}
   profile: ambient
   EOF
     

6. Verify that the components of the Istio ambient control plane are successfully installed. Because the Istio CNI is deployed as a daemon set, the number of CNI pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.

     kubectl get pods -A --context ${CLUSTER_CONTEXT} | grep istio
     

   Example output:

     istio-system   istiod-85c4dfd97f-mncj5                             1/1     Running   0               40s
   istio-system   istio-cni-node-pr5rl                                1/1     Running   0               9s
   istio-system   istio-cni-node-pvmx2                                1/1     Running   0               9s
   istio-system   istio-cni-node-6q26l                                1/1     Running   0               9s
     

7. Install the ztunnel daemonset.

     helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
   --namespace istio-system \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   configValidation: true
   enabled: true
   env:
     L7_ENABLED: "true"
     # Required when a unique trust domain is set for each cluster
     SKIP_VALIDATE_TRUST_DOMAIN: "true"
   hub: ${REPO}
   istioNamespace: istio-system
   multiCluster:
     clusterName: ${CLUSTER_NAME}
   namespace: istio-system
   network: ${CLUSTER_NAME}
   profile: ambient
   proxy:
     clusterDomain: cluster.local
   tag: ${ISTIO_IMAGE}
   terminationGracePeriodSeconds: 29
   variant: distroless
   EOF
     

     helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
   --namespace kube-system \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   configValidation: true
   enabled: true
   env:
     L7_ENABLED: "true"
     # Required when a unique trust domain is set for each cluster
     SKIP_VALIDATE_TRUST_DOMAIN: "true"
   global:
     platform: openshift
   hub: ${REPO}
   istioNamespace: istio-system
   multiCluster:
     clusterName: ${CLUSTER_NAME}
   namespace: kube-system
   network: ${CLUSTER_NAME}
   profile: ambient
   proxy:
     clusterDomain: cluster.local
   tag: ${ISTIO_IMAGE}
   terminationGracePeriodSeconds: 29
   variant: distroless
   EOF
     

8. Verify that the ztunnel pods are successfully installed. Because the ztunnel is deployed as a daemon set, the number of pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.

     kubectl get pods -A --context ${CLUSTER_CONTEXT} | grep ztunnel
     

   Example output:

     ztunnel-tvtzn             1/1     Running   0          7s
   ztunnel-vtpjm             1/1     Running   0          4s
   ztunnel-hllxg             1/1     Running   0          4s
     

9. Label the istio-system namespace with the cluster’s network name, which you previously set to your cluster’s name in the global.network field of the istiod installation. The ambient control plane uses this label internally to group pods that exist in the same L3 network.

     kubectl label namespace istio-system --context ${CLUSTER_CONTEXT} topology.istio.io/network=${CLUSTER_NAME}
     

10. Create an east-west gateway in the istio-eastwest namespace. An east-west gateway facilitates traffic between services in each cluster in your multicluster mesh.

    • You can use the following istioctl command to quickly create the east-west gateway.

        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
      istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT}
        

    • To take a look at the Gateway resource that this command creates, you can include the --generate flag in the command.

        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
      istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT} --generate
        

      In this example output, the gatewayClassName that is used, istio-eastwest, is included by default when you install Istio in ambient mode.

        apiVersion: gateway.networking.k8s.io/v1
      kind: Gateway
      metadata:
        labels:
          istio.io/expose-istiod: "15012"
          topology.istio.io/network: "<cluster_network_name>"
        name: istio-eastwest
        namespace: istio-eastwest
      spec:
        gatewayClassName: istio-eastwest
        listeners:
        - name: cross-network
          port: 15008
          protocol: HBONE
          tls:
            mode: Passthrough
        - name: xds-tls
          port: 15012
          protocol: TLS
          tls:
            mode: Passthrough
        

11. Verify that the east-west gateway is successfully deployed.

      kubectl get pods -n istio-eastwest --context $CLUSTER_CONTEXT
      

12. For each cluster that you want to include in the multicluster ambient mesh setup, repeat these steps to install the Gloo Operator, ambient mesh components, and east-west gateway in each cluster. Remember to change the cluster name and context variables each time you repeat the steps.

      export CLUSTER_NAME=<cluster-name>
    export CLUSTER_CONTEXT=<cluster-context>
      

Link clusters link

Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters.

1. Verify that the contexts for the clusters that you want to include in the multicluster mesh are listed in your kubeconfig file.

     kubectl config get-contexts
     

   • In the output, note the names of the cluster contexts, which you use in the next step to link the clusters.
   • If you have multiple kubeconfig files, you can generate a merged kubeconfig file by running the following command.

       KUBECONFIG=<kubeconfig_file1>.yaml:<file2>.yaml:<file3>.yaml kubectl config view --flatten
       

2. Using the names of the cluster contexts, link the clusters so that they can communicate. Note that you can either link the clusters bi-directionally or asymmetrically. In a standard bi-directional setup, services in any of the linked clusters can send requests to and receive requests from the services in any of the other linked clusters. In an asymmetrical setup, you allow one cluster to send requests to another cluster, but the other cluster cannot send requests back to the first cluster.

   You can use the following istioctl command to quickly link the clusters bi-directionally. In each cluster, Gateway resources are created that use the istio-remote GatewayClass. This class allows the gateways to connect to other clusters by using the clusters’ contexts.

     istioctl multicluster link --namespace istio-eastwest --contexts=<context1>,<context2>,<context3>
     

   To take a look at the Gateway resources that this command creates, you can include the --generate flag in the command.

   Example output for two clusters:

     Gateway istio-eastwest/istio-remote-peer-cluster1 applied to cluster "<cluster2_context>" pointing to cluster "<cluster1_context>" (network "cluster1")
   Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "<cluster1_context>" pointing to cluster "<cluster2_context>" (network "cluster2")
     

   You can use the following istioctl command to quickly link the clusters asymmetrically. The services in the cluster in the --from flag can send requests to services in cluster in the --to flag, but sending requests in the reverse direction is not permitted.

     istioctl multicluster link --namespace istio-eastwest --from <context1> --to <context2>
     

   To take a look at the Gateway resources that this command creates, you can include the --generate flag in the command.

   For example, this command allows services in cluster1’s mesh to send requests to services in cluster2’s mesh through cluster2’s east-west gateway. However, the reverse is not permitted: services in cluster2’s mesh cannot send requests through cluster1’s east-west gateway to services in cluster1.

     istioctl multicluster link --namespace istio-eastwest --from cluster1 --to cluster2
     

   Example output:

     Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "<cluster1_context>" pointing to cluster "<cluster2_context>" (network "cluster2")
     

  kubectl get gateways -n istio-eastwest --context $CLUSTER_CONTEXT
  

  NAME                                     CLASS            ADDRESS     PROGRAMMED   AGE
istio-eastwest                           istio-eastwest   <address>   True         29m
istio-remote-peer-cluster1               istio-remote     <address>   True         16m
...
  

  kubectl get svc -n istio-eastwest --context $CLUSTER_CONTEXT
  

  NAME             TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                           AGE
istio-eastwest   LoadBalancer   172.20.44.255   <address>       15021:31730/TCP,15008:30632/TCP,15012:32356/TCP   29m
  

Automatically link clusters (beta) link

In each cluster, use Helm to create the ambient mesh components, and create an east-west gateway so that traffic requests can be routed cross-cluster. Then, use the Gloo Mesh management plane to automate multicluster linking, which enables cross-cluster service discovery.

Set up tools link

1. Set your Enterprise level license for Gloo Mesh as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing.

     export GLOO_MESH_LICENSE_KEY=<enterprise_license_key>
     

2. Choose the version of Istio that you want to install or upgrade to by reviewing the supported versions. In Gloo Mesh version 2.7 and later, multicluster setups require version 1.24.3 or later.

3. Save the details for the version of the Solo distribution of Istio that you want to install.

     export ISTIO_VERSION=1.25.2
   export ISTIO_IMAGE=${ISTIO_VERSION}-solo
   export REPO_KEY=e038d180f90a
     

   1. Save the Solo distribution of Istio patch version.

        export ISTIO_VERSION=1.24.5
      export ISTIO_IMAGE=${ISTIO_VERSION}-solo
        

   2. Save the repo key for version 1.24 of the Solo distribution of Istio. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Ambient section of the Istio images built by Solo.io support article.

        # 12-character hash at the end of the 1.24 repo URL
      export REPO_KEY=<repo_key>
        

4. Get the Solo distribution of Istio binary and install istioctl, which you use for multicluster linking and gateway commands.

   1. Get the OS and architecture that you use on your machine.

        OS=$(uname | tr '[:upper:]' '[:lower:]' | sed -E 's/darwin/osx/')
      ARCH=$(uname -m | sed -E 's/aarch/arm/; s/x86_64/amd64/; s/armv7l/armv7/')
      echo $OS
      echo $ARCH
        

   2. Download the Solo distribution of Istio binary and install istioctl.

        mkdir -p ~/.istioctl/bin
      curl -sSL https://storage.googleapis.com/istio-binaries-$REPO_KEY/$ISTIO_IMAGE/istioctl-$ISTIO_IMAGE-$OS-$ARCH.tar.gz | tar xzf - -C ~/.istioctl/bin
      chmod +x ~/.istioctl/bin/istioctl
      
      export PATH=${HOME}/.istioctl/bin:${PATH}
        

   3. Verify that the istioctl client runs the Solo distribution of Istio that you want to install.

        istioctl version --remote=false
        

      Example output:

        client version: 1.25.2-solo
        

Enable automatic peering of clusters link

Upgrade Gloo Mesh in your multicluster setup to enable the ConfigDistribution feature flag and install the enterprise CRDs, which are required for Gloo Mesh to automate peering and distribute gateways between clusters.

1. Upgrade your gloo-platform-crds Helm release in the management cluster to include the following settings.

     helm get values gloo-platform-crds -n gloo-mesh -o yaml --kube-context ${MGMT_CONTEXT} > mgmt-crds.yaml
   helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \
       --kube-context ${MGMT_CONTEXT} \
       --namespace gloo-mesh \
       -f mgmt-crds.yaml \
       --set featureGates.ConfigDistribution=true \
       --set installEnterpriseCrds=true
     

2. Upgrade your gloo-platform Helm release in the management cluster to include the following settings.

     helm get values gloo-platform -n gloo-mesh -o yaml --kube-context ${MGMT_CONTEXT} > mgmt-plane.yaml
   helm upgrade gloo-platform gloo-platform/gloo-platform \
       --kube-context ${MGMT_CONTEXT} \
       --namespace gloo-mesh \
       -f mgmt-plane.yaml \
       --set featureGates.ConfigDistribution=true
     

3. Upgrade your gloo-platform-crds Helm release in each workload cluster to include the following settings. Repeat this step for each workload cluster.

     helm get values gloo-platform-crds -n gloo-mesh -o yaml --kube-context ${CLUSTER_CONTEXT} > crds.yaml
   helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \
       --kube-context ${CLUSTER_CONTEXT} \
       --namespace gloo-mesh \
       -f crds.yaml \
       --set installEnterpriseCrds=true
     

Create a shared root of trust link

Create a shared root of trust for each cluster in the multicluster setup, including the management cluster. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.

  curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}

mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

function create_cacerts_secret() {
  context=${1:?context}
  cluster=${2:?cluster}
  make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
  kubectl --context=${context} create ns istio-system || true
  kubectl --context=${context} create secret generic cacerts -n istio-system \
    --from-file=${cluster}/ca-cert.pem \
    --from-file=${cluster}/ca-key.pem \
    --from-file=${cluster}/root-cert.pem \
    --from-file=${cluster}/cert-chain.pem
}

create_cacerts_secret ${MGMT_CONTEXT} ${MGMT_CLUSTER}
create_cacerts_secret ${REMOTE_CONTEXT1} ${REMOTE_CLUSTER1}
create_cacerts_secret ${REMOTE_CONTEXT2} ${REMOTE_CLUSTER2}

cd ../..
  

Deploy ambient components link

1. Save the name and kubeconfig context of a cluster where you want to install Istio in the following environment variables. Each time you repeat the steps in this guide, you change these variables to the next workload cluster’s name and context. Note that to use automated multicluster peering, you must complete these steps to install an ambient mesh in the management cluster as well as your workload clusters.

     export CLUSTER_NAME=<cluster-name>
   export CLUSTER_CONTEXT=<cluster-context>
     

2. Apply the CRDs for the Kubernetes Gateway API to your cluster, which are required to create components such as waypoint proxies for L7 traffic policies, gateways with the Gateway resource, and more.

     kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v/standard-install.yaml --context ${CLUSTER_CONTEXT}
     

3. Install the base chart, which contains the CRDs and cluster roles required to set up Istio.

     helm upgrade --install istio-base oci://${HELM_REPO}/base \
   --namespace istio-system \
   --create-namespace \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   defaultRevision: ""
   profile: ambient
   EOF
     

     helm upgrade --install istio-base oci://${HELM_REPO}/base \
   --namespace istio-system \
   --create-namespace \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   defaultRevision: ""
   profile: ambient
   global:
     platform: openshift
   EOF
     

4. Create the istiod control plane in your cluster.

   • Istio 1.25 and later: If you prefer to specify your license secret instead of an inline value, you can uncomment the secretRef fields.

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     env:
       # Enables automatic creation of remote peer gateways
       PEERING_AUTOMATIC_LOCAL_GATEWAY: "true"
       # Assigns IP addresses to multicluster services
       PILOT_ENABLE_IP_AUTOALLOCATE: "true"
       # Disable selecting workload entries for local service routing.
       # Required for Gloo VirtualDestinaton functionality.
       PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
       # Required when meshConfig.trustDomain is set
       PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
     global:
       hub: ${REPO}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
     istio_cni:
       namespace: istio-system
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       # Assign each cluster a unique trust domain to apply policies to specific clusters
       trustDomain: "${CLUSTER_NAME}.local"
     # Required to enable multicluster support
     platforms:
       peering:
         enabled: true
     profile: ambient
     license:
       value: ${GLOO_MESH_LICENSE_KEY}
       # secretRef:
       #   name: 
       #   namespace: 
     EOF
       

   • Istio 1.24 and earlier:

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     env:
       # Assigns IP addresses to multicluster services
       PILOT_ENABLE_IP_AUTOALLOCATE: "true"
       # Disable selecting workload entries for local service routing.
       # Required for Gloo VirtualDestinaton functionality.
       PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
       # Required when meshConfig.trustDomain is set
       PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
     global:
       hub: ${REPO}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
     istio_cni:
       namespace: istio-system
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       # Assign each cluster a unique trust domain to apply policies to specific clusters
       trustDomain: "${CLUSTER_NAME}.local"
     # Required to enable multicluster support
     platforms:
       peering:
         enabled: true
     profile: ambient
     EOF
       

   • Istio 1.25 and later: If you prefer to specify your license secret instead of an inline value, you can uncomment the secretRef fields.

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     env:
       # Assigns IP addresses to multicluster services
       PILOT_ENABLE_IP_AUTOALLOCATE: "true"
       # Disable selecting workload entries for local service routing.
       # Required for Gloo VirtualDestinaton functionality.
       PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
       # Required when meshConfig.trustDomain is set
       PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
     global:
       hub: ${REPO}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
       platform: openshift
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
     istio_cni:
       namespace: kube-system
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       # Assign each cluster a unique trust domain to apply policies to specific clusters
       trustDomain: "${CLUSTER_NAME}.local"
     # Required to enable multicluster support
     platforms:
       peering:
         enabled: true
     profile: ambient
     license:
       value: ${GLOO_MESH_LICENSE_KEY}
       # secretRef:
       #   name: 
       #   namespace: 
     EOF
       

   • Istio 1.24 and earlier:

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     env:
       # Assigns IP addresses to multicluster services
       PILOT_ENABLE_IP_AUTOALLOCATE: "true"
       # Disable selecting workload entries for local service routing.
       # Required for Gloo VirtualDestinaton functionality.
       PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
       # Required when meshConfig.trustDomain is set
       PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
     global:
       hub: ${REPO}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
       platform: openshift
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
     istio_cni:
       namespace: kube-system
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       # Assign each cluster a unique trust domain to apply policies to specific clusters
       trustDomain: "${CLUSTER_NAME}.local"
     # Required to enable multicluster support
     platforms:
       peering:
         enabled: true
     profile: ambient
     EOF
       

5. Install the Istio CNI node agent daemonset. Note that although the CNI is included in this section, it is technically not part of the control plane or data plane.

     helm upgrade --install istio-cni oci://${HELM_REPO}/cni \
   --namespace istio-system \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   # Assigns IP addresses to multicluster services
   ambient:
     dnsCapture: true
   excludeNamespaces:
     - istio-system
     - kube-system
   global:
     hub: ${REPO}
     tag: ${ISTIO_IMAGE}
   profile: ambient
   EOF
     

     helm upgrade --install istio-cni oci://${HELM_REPO}/cni \
   --namespace kube-system \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   # Assigns IP addresses to multicluster services
   ambient:
     dnsCapture: true
   excludeNamespaces:
     - istio-system
     - kube-system
   global:
     hub: ${REPO}
     platform: openshift
     tag: ${ISTIO_IMAGE}
   profile: ambient
   EOF
     

6. Verify that the components of the Istio ambient control plane are successfully installed. Because the Istio CNI is deployed as a daemon set, the number of CNI pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.

     kubectl get pods -A --context ${CLUSTER_CONTEXT} | grep istio
     

   Example output:

     istio-system   istiod-85c4dfd97f-mncj5                             1/1     Running   0               40s
   istio-system   istio-cni-node-pr5rl                                1/1     Running   0               9s
   istio-system   istio-cni-node-pvmx2                                1/1     Running   0               9s
   istio-system   istio-cni-node-6q26l                                1/1     Running   0               9s
     

7. Install the ztunnel daemonset.

     helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
   --namespace istio-system \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   configValidation: true
   enabled: true
   env:
     L7_ENABLED: "true"
     # Required when a unique trust domain is set for each cluster
     SKIP_VALIDATE_TRUST_DOMAIN: "true"
   hub: ${REPO}
   istioNamespace: istio-system
   multiCluster:
     clusterName: ${CLUSTER_NAME}
   namespace: istio-system
   network: ${CLUSTER_NAME}
   profile: ambient
   proxy:
     clusterDomain: cluster.local
   tag: ${ISTIO_IMAGE}
   terminationGracePeriodSeconds: 29
   variant: distroless
   EOF
     

     helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
   --namespace kube-system \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   configValidation: true
   enabled: true
   env:
     L7_ENABLED: "true"
     # Required when a unique trust domain is set for each cluster
     SKIP_VALIDATE_TRUST_DOMAIN: "true"
   global:
     platform: openshift
   hub: ${REPO}
   istioNamespace: istio-system
   multiCluster:
     clusterName: ${CLUSTER_NAME}
   namespace: kube-system
   network: ${CLUSTER_NAME}
   profile: ambient
   proxy:
     clusterDomain: cluster.local
   tag: ${ISTIO_IMAGE}
   terminationGracePeriodSeconds: 29
   variant: distroless
   EOF
     

8. Verify that the ztunnel pods are successfully installed. Because the ztunnel is deployed as a daemon set, the number of pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.

     kubectl get pods -A --context ${CLUSTER_CONTEXT} | grep ztunnel
     

   Example output:

     ztunnel-tvtzn             1/1     Running   0          7s
   ztunnel-vtpjm             1/1     Running   0          4s
   ztunnel-hllxg             1/1     Running   0          4s
     

9. Label the istio-system namespace with the cluster’s network name, which you previously set to your cluster’s name in the global.network field of the istiod installation. The ambient control plane uses this label internally to group pods that exist in the same L3 network.

     kubectl label namespace istio-system --context ${CLUSTER_CONTEXT} topology.istio.io/network=${CLUSTER_NAME}
     

10. Create an east-west gateway in the istio-eastwest namespace. An east-west gateway facilitates traffic between services in each cluster in your multicluster mesh.

    • You can use the following istioctl command to quickly create the east-west gateway.

        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
      istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT}
        

    • To take a look at the Gateway resource that this command creates, you can include the --generate flag in the command.

        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
      istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT} --generate
        

      In this example output, the gatewayClassName that is used, istio-eastwest, is included by default when you install Istio in ambient mode.

        apiVersion: gateway.networking.k8s.io/v1
      kind: Gateway
      metadata:
        labels:
          istio.io/expose-istiod: "15012"
          topology.istio.io/network: "<cluster_network_name>"
        name: istio-eastwest
        namespace: istio-eastwest
      spec:
        gatewayClassName: istio-eastwest
        listeners:
        - name: cross-network
          port: 15008
          protocol: HBONE
          tls:
            mode: Passthrough
        - name: xds-tls
          port: 15012
          protocol: TLS
          tls:
            mode: Passthrough
        

11. Verify that the east-west gateway is successfully deployed.

      kubectl get pods -n istio-eastwest --context $CLUSTER_CONTEXT
      

12. For each cluster that you want to include in the multicluster ambient mesh setup, including the management cluster, repeat these steps to install the ambient mesh components and an east-west gateway in each cluster. Remember to change the cluster name and context variables each time you repeat the steps.

      export CLUSTER_NAME=<cluster-name>
    export CLUSTER_CONTEXT=<cluster-context>
      

Review remote peer gateways link

After you complete the steps for each cluster, verify that Gloo Mesh successfully created and distributed the remote peering gateways. These gateways use the istio-remote GatewayClass, which allows the istiod control plane in each cluster to discover the east-west gateway addresses of other clusters. Gloo Mesh generates one istio-remote resource in the management cluster for each connected workload cluster, and then distributes the gateway to each cluster respectively.

1. Verify that an istio-remote gateway for each connected cluster is copied to the management cluster.

     kubectl get gateways -n istio-eastwest --context $MGMT_CONTEXT
     

   In this example output, the istio-remote gateways that were auto-generated for workload clusters cluster1 and cluster2 are copied to the management cluster, alongside the management cluster’s own istio-remote gateway and east-west gateway.

     NAMESPACE        NAME                            CLASS           ADDRESS                                                                   PROGRAMMED   AGE
   istio-eastwest   istio-eastwest                 istio-eastwest   a7f6f1a2611fc4eb3864f8d688622fd4-1234567890.us-east-1.elb.amazonaws.com   True         6s
   istio-eastwest   istio-remote-peer-cluster1     istio-remote     a5082fe9522834b8192a6513eb8c6b01-0987654321.us-east-1.elb.amazonaws.com   True         4s
   istio-eastwest   istio-remote-peer-cluster2     istio-remote     aaad62dc3ffb142a1bfc13df7fe9665b-5678901234.us-east-1.elb.amazonaws.com   True         4s
   istio-eastwest   istio-remote-peer-mgmt         istio-remote     a7f6f1a2611fc4eb3864f8d688622fd4-1234567890.us-east-1.elb.amazonaws.com   True         4s
     

2. In each cluster, verify that all istio-remote gateways are successfully distributed to all workload clusters. This ensures that services in each workload cluster can now access the east-west gateways in other clusters of the multicluster mesh setup.

     kubectl get gateways -n istio-eastwest --context $CLUSTER_CONTEXT
     

Next link

• Add apps to the ambient mesh. For multicluster setups, this includes making specific services available across your linked cluster setup.
• In a multicluster mesh, the east-west gateway serves as a ztunnel that allows traffic requests to flow across clusters, but it does not modify requests in any way. To control in-mesh traffic, you can instead apply policies to waypoint proxies that you create for a workload namespace.