Create a multicluster mesh with Helm

Considerations link

Before you install a multicluster sidecar mesh, review the following considerations and requirements.

License requirements

Version requirements

In Gloo Mesh version 2.7 and later, multicluster setups require the Solo distribution of Istio version 1.24.3 or later (1.24.3-solo), including the Solo distribution of istioctl.

Components

In the following steps, you install the Istio ambient components in each workload cluster to successfully create east-west gateways and establish multicluster peering, even if you plan to use a sidecar mesh. However, sidecar mesh setups continue to use sidecar injection for your workloads. Your workloads are not added to an ambient mesh. For more information about running both ambient and sidecar components in one mesh setup, see Ambient-sidecar interoperability.

Revision and canary upgrade limitations

The upgrade guides in this documentation show you how to perform in-place upgrades for your Istio components, which is the recommended upgrade strategy.

Set up tools link

1. Save the following environment details and install the Solo distribution of the istioctl binary.

   1. Choose the version of Istio that you want to install or upgrade to by reviewing the supported versions. In Gloo Mesh version 2.7 and later, multicluster setups require version 1.24.3 or later.

   2. Decide on the specific tag of Solo distribution of Istio image, such as -solo, -solo-fips, -solo-distroless, or -solo-fips-distroless, that you want for your environment.

   3. Save the details for the version of the Solo distribution of Istio that you want to install.

      1. Save the details for version 1.25 of the Solo distribution of Istio.

              # Change the tags as needed
            export ISTIO_IMAGE=1.25.2-solo
            export REPO_KEY=e038d180f90a
            export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
            export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}
              

      2. Set your Enterprise level license for Gloo Mesh as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing.

              export GLOO_MESH_LICENSE_KEY=<enterprise_license_key>
              

      notifications

      In the Solo distribution of Istio 1.25 and later, you can access enterprise-level features by passing your Solo license in the license.value or license.secretRef field of your Solo istiod Helm chart. The Helm chart that is provided by Solo includes safeguards, default settings, and upgrade handling to ensure a reliable and secure Istio deployment. Although you can pass the license key in the pilot.env.SOLO_LICENSE_KEY field of the open source Istio Helm chart, this method is not recommended. For best results, we strongly recommend using the Solo Helm chart to install and manage Istio. For example, to install the Solo distribution of 1.25, you can run the helm install istiod oci://us-docker.pkg.dev/gloo-mesh/istio-helm-e038d180f90a/istiod ... command and provide your installation values in --set flags or a -f <file.yaml> config file.

      1. Save the Solo distribution of Istio patch version and tag.

              # Change the tags as needed
            export ISTIO_IMAGE=1.24.5-solo
              

      2. Save the repo key for version 1.24 of the Solo distribution of Istio. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Ambient section of the Istio images built by Solo.io support article.

              # 12-character hash at the end of the 1.24 repo URL
            export REPO_KEY=<repo_key>
            export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
            export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}
              

   4. Get the Solo distribution of Istio binary and install istioctl, which you use for multicluster linking and gateway commands.

      1. Get the OS and architecture that you use on your machine.

              OS=$(uname | tr '[:upper:]' '[:lower:]' | sed -E 's/darwin/osx/')
            ARCH=$(uname -m | sed -E 's/aarch/arm/; s/x86_64/amd64/; s/armv7l/armv7/')
            echo $OS
            echo $ARCH
              

      2. Download the Solo distribution of Istio binary and install istioctl.

              mkdir -p ~/.istioctl/bin
            curl -sSL https://storage.googleapis.com/istio-binaries-$REPO_KEY/$ISTIO_IMAGE/istioctl-$ISTIO_IMAGE-$OS-$ARCH.tar.gz | tar xzf - -C ~/.istioctl/bin
            chmod +x ~/.istioctl/bin/istioctl
            
            export PATH=${HOME}/.istioctl/bin:${PATH}
              

      3. Verify that the istioctl client runs the Solo distribution of Istio that you want to install.

              istioctl version --remote=false
              

         Example output:

              client version: 1.25.2-solo
              

   5. If you use Google Kubernetes Engine (GKE) clusters, create the following ResourceQuota in the istio-system namespace of each cluster. For more information about this requirement, see the community Istio documentation.

           kubectl create namespace istio-system
         kubectl -n istio-system apply -f - <<EOF
         apiVersion: v1
         kind: ResourceQuota
         metadata:
           name: gcp-critical-pods
           namespace: istio-system
         spec:
           hard:
             pods: 1000
           scopeSelector:
             matchExpressions:
             - operator: In
               scopeName: PriorityClass
               values:
               - system-node-critical
         EOF
           

2. Deploy Istio to each cluster and link clusters. These steps vary based on whether you want to use Gloo Mesh to automatically link clusters or manually link clusters yourself. Note that automatic linking is a beta feature, and requires Istio to be installed in the same cluster that the Gloo Mesh management plane is deployed to.

Manually link clusters link

In each cluster, use Helm to create the service mesh components. Then, create an east-west gateway so that traffic requests can be routed cross-cluster, and link clusters to enable cross-cluster service discovery.

Create a shared root of trust link

Each cluster in the multicluster setup must have a shared root of trust. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.

  curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}

mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

function create_cacerts_secret() {
  context=${1:?context}
  cluster=${2:?cluster}
  make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
  kubectl --context=${context} create ns istio-system || true
  kubectl --context=${context} create secret generic cacerts -n istio-system \
    --from-file=${cluster}/ca-cert.pem \
    --from-file=${cluster}/ca-key.pem \
    --from-file=${cluster}/root-cert.pem \
    --from-file=${cluster}/cert-chain.pem
}

create_cacerts_secret ${REMOTE_CONTEXT1} ${REMOTE_CLUSTER1}
create_cacerts_secret ${REMOTE_CONTEXT2} ${REMOTE_CLUSTER2}

cd ../..
  

Deploy mesh components link

1. Save the name and kubeconfig context of a cluster where you want to install Istio in the following environment variables. Each time you repeat the steps in this guide, you change these variables to the next workload cluster’s name and context.

     export CLUSTER_NAME=<cluster-name>
   export CLUSTER_CONTEXT=<cluster-context>
     

2. Apply the CRDs for the Kubernetes Gateway API to your cluster, which are required to create components such as waypoint proxies for L7 traffic policies, gateways with the Gateway resource, and more.

     kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v/standard-install.yaml --context ${CLUSTER_CONTEXT}
     

3. Install the base chart, which contains the CRDs and cluster roles required to set up Istio.

     helm upgrade --install istio-base oci://${HELM_REPO}/base \
   --namespace istio-system \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   --create-namespace \
   -f - <<EOF
   profile: ambient
   revision: main
   global:
     istioNamespace: istio-system
   EOF
     

     helm upgrade --install istio-base oci://${HELM_REPO}/base \
   --namespace istio-system \
   --create-namespace \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   profile: ambient
   revision: main
   global:
     istioNamespace: istio-system
     platform: openshift
   EOF
     

4. Create the istiod control plane in your cluster.

   • Istio 1.25 and later: If you prefer to specify your license secret instead of an inline value, you can uncomment the secretRef fields.

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     global:
       hub: ${REPO}
       variant: distroless
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
     profile: ambient
     revision: main
     istio_cni:
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       trustDomain: ${CLUSTER_NAME}
     pilot:
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
             - podAffinityTerm:
                 labelSelector:
                   matchExpressions:
                   - key: istio.io/rev
                     operator: In
                     values:
                     - main
                 topologyKey: topology.kubernetes.io/zone
               weight: 100
       enabled: true
       env:
         PILOT_ENABLE_IP_AUTOALLOCATE: "true"
         PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
         PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
         AUTO_RELOAD_PLUGIN_CERTS: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS: "true"
     platforms:
       peering:
         enabled: true
     revisionTags:
     - default
     license:
       value: ${GLOO_MESH_LICENSE_KEY}
       # secretRef:
       #   name: 
       #   namespace: 
     EOF
       

   • Istio 1.24 and earlier:

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     global:
       hub: ${REPO}
       variant: distroless
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
     profile: ambient
     revision: main
     istio_cni:
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       trustDomain: ${CLUSTER_NAME}
     pilot:
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
             - podAffinityTerm:
                 labelSelector:
                   matchExpressions:
                   - key: istio.io/rev
                     operator: In
                     values:
                     - main
                 topologyKey: topology.kubernetes.io/zone
               weight: 100
       enabled: true
       env:
         PILOT_ENABLE_IP_AUTOALLOCATE: "true"
         PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
         PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
         AUTO_RELOAD_PLUGIN_CERTS: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS: "true"
     platforms:
       peering:
         enabled: true
     revisionTags:
     - default
     EOF
       

   • Istio 1.25 and later: If you prefer to specify your license secret instead of an inline value, you can uncomment the secretRef fields.

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     global:
       hub: ${REPO}
       variant: distroless
       platform: openshift
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
     profile: ambient
     revision: main
     istio_cni:
       enabled: true
       namespace: kube-system
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       trustDomain: ${CLUSTER_NAME}
     pilot:
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
             - podAffinityTerm:
                 labelSelector:
                   matchExpressions:
                   - key: istio.io/rev
                     operator: In
                     values:
                     - main
                 topologyKey: topology.kubernetes.io/zone
               weight: 100
       enabled: true
       env:
         PILOT_ENABLE_IP_AUTOALLOCATE: "true"
         PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
         PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
         AUTO_RELOAD_PLUGIN_CERTS: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS: "true"
     platforms:
       peering:
         enabled: true
     revisionTags:
     - default
     license:
       value: ${GLOO_MESH_LICENSE_KEY}
       # secretRef:
       #   name: 
       #   namespace: 
     EOF
       

   • Istio 1.24 and earlier:

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     global:
       hub: ${REPO}
       variant: distroless
       platform: openshift
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
     profile: ambient
     revision: main
     istio_cni:
       enabled: true
       namespace: kube-system
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       trustDomain: ${CLUSTER_NAME}
     pilot:
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
             - podAffinityTerm:
                 labelSelector:
                   matchExpressions:
                   - key: istio.io/rev
                     operator: In
                     values:
                     - main
                 topologyKey: topology.kubernetes.io/zone
               weight: 100
       enabled: true
       env:
         PILOT_ENABLE_IP_AUTOALLOCATE: "true"
         PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
         PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
         AUTO_RELOAD_PLUGIN_CERTS: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS: "true"
     platforms:
       peering:
         enabled: true
     revisionTags:
     - default
     EOF
       

5. Install the Istio CNI node agent daemonset. Note that although the CNI is included in this section, it is technically not part of the control plane or data plane.

     helm upgrade --install istio-cni oci://${HELM_REPO}/cni \
   --namespace istio-system \
   --kube-context=${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   global:
     hub: ${REPO}
     proxy: ${ISTIO_IMAGE}
     variant: distroless
   profile: ambient
   revision: main
   cni:
     ambient:
       dnsCapture: true
     excludeNamespaces:
     - istio-system
     - kube-system
   EOF
     

     helm upgrade --install istio-cni oci://${HELM_REPO}/cni \
   --namespace kube-system \
   --kube-context=${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   --create-namespace \
   -f - <<EOF
   global:
     hub: ${REPO}
     proxy: ${ISTIO_IMAGE}
     variant: distroless
     platform: openshift
   profile: ambient
   revision: main
   cni:
     ambient:
       dnsCapture: true
     excludeNamespaces:
     - istio-system
     - kube-system
   EOF
     

6. Verify that the components of the Istio control plane are successfully installed. Because the Istio CNI is deployed as a daemon set, the number of CNI pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.

     kubectl get pods -A --context ${CLUSTER_CONTEXT} | grep istio
     

   Example output:

     istio-system   istio-cni-node-g84js                   1/1     Running   0          8s
   istio-system   istio-cni-node-gwssc                   1/1     Running   0          8s
   istio-system   istiod-main-7c59ff7bd7-4mc25           1/1     Running   0          19s
     

7. Install the ztunnel daemonset. Note that this component is required to successfully peer clusters together.

     helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
   --namespace istio-system \
   --kube-context=${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   configValidation: true
   enabled: true
   env:
     L7_ENABLED: "true"
     ISTIO_META_DNS_CAPTURE: "true"
     SKIP_VALIDATE_TRUST_DOMAIN: "true"
   hub: ${REPO}
   istioNamespace: istio-system
   multiCluster:
     clusterName: ${CLUSTER_NAME}
   network: ${CLUSTER_NAME}
   namespace: istio-system
   profile: ambient
   proxy:
     clusterDomain: cluster.local
   tag: ${ISTIO_IMAGE}
   variant: distroless
   revision: main
   EOF
     

     helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
   --namespace kube-system \
   --kube-context=${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   configValidation: true
   enabled: true
   env:
     L7_ENABLED: "true"
     ISTIO_META_DNS_CAPTURE: "true"
     SKIP_VALIDATE_TRUST_DOMAIN: "true"
   hub: ${REPO}
   istioNamespace: istio-system
   multiCluster:
     clusterName: ${CLUSTER_NAME}
   network: ${CLUSTER_NAME}
   namespace: kube-system
   profile: ambient
   global: 
     platform: openshift
   proxy:
     clusterDomain: cluster.local
   tag: ${ISTIO_IMAGE}
   terminationGracePeriodSeconds: 29
   variant: distroless
   revision: main
   EOF
     

8. Verify that the ztunnel pods are successfully installed. Because the ztunnel is deployed as a daemon set, the number of pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.

     kubectl get pods -A --context ${CLUSTER_CONTEXT} | grep ztunnel
     

   Example output:

     ztunnel-tvtzn             1/1     Running   0          7s
   ztunnel-vtpjm             1/1     Running   0          4s
   ztunnel-hllxg             1/1     Running   0          4s
     

9. Label the istio-system namespace with the cluster’s network name, which you previously set to your cluster’s name in the global.network field of the istiod installation. The Istio control plane uses this label internally to group pods that exist in the same L3 network.

     kubectl label namespace istio-system --context ${CLUSTER_CONTEXT} topology.istio.io/network=${CLUSTER_NAME}
     

10. Create an east-west gateway in the istio-eastwest namespace. An east-west gateway facilitates traffic between services in each cluster in your multicluster mesh.

    • You can use the following istioctl command to quickly create the east-west gateway.

        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
      istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT}
        

    • To take a look at the Gateway resource that this command creates, you can include the --generate flag in the command.

        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
      istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT} --generate
        

      In this example output, the gatewayClassName that is used, istio-eastwest, is included by default when you install Istio in ambient mode.

        apiVersion: gateway.networking.k8s.io/v1
      kind: Gateway
      metadata:
        labels:
          istio.io/expose-istiod: "15012"
          topology.istio.io/network: "<cluster_network_name>"
        name: istio-eastwest
        namespace: istio-eastwest
      spec:
        gatewayClassName: istio-eastwest
        listeners:
        - name: cross-network
          port: 15008
          protocol: HBONE
          tls:
            mode: Passthrough
        - name: xds-tls
          port: 15012
          protocol: TLS
          tls:
            mode: Passthrough
        

11. Verify that the east-west gateway is successfully deployed.

      kubectl get pods -n istio-eastwest --context $CLUSTER_CONTEXT
      

12. For each cluster that you want to include in the multicluster service mesh setup, repeat these steps to install the Gloo Operator, service mesh components, and east-west gateway in each cluster. Remember to change the cluster name and context variables each time you repeat the steps.

      export CLUSTER_NAME=<cluster-name>
    export CLUSTER_CONTEXT=<cluster-context>
      

Link clusters link

Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters.

1. Verify that the contexts for the clusters that you want to include in the multicluster mesh are listed in your kubeconfig file.

     kubectl config get-contexts
     

   • In the output, note the names of the cluster contexts, which you use in the next step to link the clusters.
   • If you have multiple kubeconfig files, you can generate a merged kubeconfig file by running the following command.

       KUBECONFIG=<kubeconfig_file1>.yaml:<file2>.yaml:<file3>.yaml kubectl config view --flatten
       

2. Using the names of the cluster contexts, link the clusters so that they can communicate. Note that you can either link the clusters bi-directionally or asymmetrically. In a standard bi-directional setup, services in any of the linked clusters can send requests to and receive requests from the services in any of the other linked clusters. In an asymmetrical setup, you allow one cluster to send requests to another cluster, but the other cluster cannot send requests back to the first cluster.

   You can use the following istioctl command to quickly link the clusters bi-directionally. In each cluster, Gateway resources are created that use the istio-remote GatewayClass. This class allows the gateways to connect to other clusters by using the clusters’ contexts.

     istioctl multicluster link --namespace istio-eastwest --contexts=<context1>,<context2>,<context3>
     

   To take a look at the Gateway resources that this command creates, you can include the --generate flag in the command.

   Example output for two clusters:

     Gateway istio-eastwest/istio-remote-peer-cluster1 applied to cluster "<cluster2_context>" pointing to cluster "<cluster1_context>" (network "cluster1")
   Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "<cluster1_context>" pointing to cluster "<cluster2_context>" (network "cluster2")
     

   You can use the following istioctl command to quickly link the clusters asymmetrically. The services in the cluster in the --from flag can send requests to services in cluster in the --to flag, but sending requests in the reverse direction is not permitted.

     istioctl multicluster link --namespace istio-eastwest --from <context1> --to <context2>
     

   To take a look at the Gateway resources that this command creates, you can include the --generate flag in the command.

   For example, this command allows services in cluster1’s mesh to send requests to services in cluster2’s mesh through cluster2’s east-west gateway. However, the reverse is not permitted: services in cluster2’s mesh cannot send requests through cluster1’s east-west gateway to services in cluster1.

     istioctl multicluster link --namespace istio-eastwest --from cluster1 --to cluster2
     

   Example output:

     Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "<cluster1_context>" pointing to cluster "<cluster2_context>" (network "cluster2")
     

  kubectl get gateways -n istio-eastwest --context $CLUSTER_CONTEXT
  

  NAME                                     CLASS            ADDRESS     PROGRAMMED   AGE
istio-eastwest                           istio-eastwest   <address>   True         29m
istio-remote-peer-cluster1               istio-remote     <address>   True         16m
...
  

  kubectl get svc -n istio-eastwest --context $CLUSTER_CONTEXT
  

  NAME             TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                           AGE
istio-eastwest   LoadBalancer   172.20.44.255   <address>       15021:31730/TCP,15008:30632/TCP,15012:32356/TCP   29m
  

Automatically link clusters (beta) link

In each cluster, use Helm to create the service mesh components, and create an east-west gateway so that traffic requests can be routed cross-cluster. Then, use the Gloo Mesh management plane to automate multicluster linking, which enables cross-cluster service discovery.

Enable automatic peering of clusters link

Upgrade Gloo Mesh in your multicluster setup to enable the ConfigDistribution feature flag and install the enterprise CRDs, which are required for Gloo Mesh to automate peering and distribute gateways between clusters.

1. Upgrade your gloo-platform-crds Helm release in the management cluster to include the following settings.

     helm get values gloo-platform-crds -n gloo-mesh -o yaml --kube-context ${MGMT_CONTEXT} > mgmt-crds.yaml
   helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \
       --kube-context ${MGMT_CONTEXT} \
       --namespace gloo-mesh \
       -f mgmt-crds.yaml \
       --set featureGates.ConfigDistribution=true \
       --set installEnterpriseCrds=true
     

2. Upgrade your gloo-platform Helm release in the management cluster to include the following settings.

     helm get values gloo-platform -n gloo-mesh -o yaml --kube-context ${MGMT_CONTEXT} > mgmt-plane.yaml
   helm upgrade gloo-platform gloo-platform/gloo-platform \
       --kube-context ${MGMT_CONTEXT} \
       --namespace gloo-mesh \
       -f mgmt-plane.yaml \
       --set featureGates.ConfigDistribution=true
     

3. Upgrade your gloo-platform-crds Helm release in each workload cluster to include the following settings. Repeat this step for each workload cluster.

     helm get values gloo-platform-crds -n gloo-mesh -o yaml --kube-context ${CLUSTER_CONTEXT} > crds.yaml
   helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \
       --kube-context ${CLUSTER_CONTEXT} \
       --namespace gloo-mesh \
       -f crds.yaml \
       --set installEnterpriseCrds=true
     

Create a shared root of trust link

Create a shared root of trust for each cluster in the multicluster setup, including the management cluster. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.

  curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}

mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

function create_cacerts_secret() {
  context=${1:?context}
  cluster=${2:?cluster}
  make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
  kubectl --context=${context} create ns istio-system || true
  kubectl --context=${context} create secret generic cacerts -n istio-system \
    --from-file=${cluster}/ca-cert.pem \
    --from-file=${cluster}/ca-key.pem \
    --from-file=${cluster}/root-cert.pem \
    --from-file=${cluster}/cert-chain.pem
}

create_cacerts_secret ${MGMT_CONTEXT} ${MGMT_CLUSTER}
create_cacerts_secret ${REMOTE_CONTEXT1} ${REMOTE_CLUSTER1}
create_cacerts_secret ${REMOTE_CONTEXT2} ${REMOTE_CLUSTER2}

cd ../..
  

Deploy mesh components link

1. Save the name and kubeconfig context of a cluster where you want to install Istio in the following environment variables. Each time you repeat the steps in this guide, you change these variables to the next workload cluster’s name and context. Note that to use automated multicluster peering, you must complete these steps to install a service mesh in the management cluster as well as your workload clusters.

     export CLUSTER_NAME=<cluster-name>
   export CLUSTER_CONTEXT=<cluster-context>
     

2. Apply the CRDs for the Kubernetes Gateway API to your cluster, which are required to create components such as waypoint proxies for L7 traffic policies, gateways with the Gateway resource, and more.

     kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v/standard-install.yaml --context ${CLUSTER_CONTEXT}
     

3. Install the base chart, which contains the CRDs and cluster roles required to set up Istio.

     helm upgrade --install istio-base oci://${HELM_REPO}/base \
   --namespace istio-system \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   --create-namespace \
   -f - <<EOF
   profile: ambient
   revision: main
   global:
     istioNamespace: istio-system
   EOF
     

     helm upgrade --install istio-base oci://${HELM_REPO}/base \
   --namespace istio-system \
   --create-namespace \
   --kube-context ${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   profile: ambient
   revision: main
   global:
     istioNamespace: istio-system
     platform: openshift
   EOF
     

4. Create the istiod control plane in your cluster.

   • Istio 1.25 and later: If you prefer to specify your license secret instead of an inline value, you can uncomment the secretRef fields.

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     global:
       hub: ${REPO}
       variant: distroless
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
     profile: ambient
     revision: main
     istio_cni:
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       trustDomain: ${CLUSTER_NAME}
     pilot:
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
             - podAffinityTerm:
                 labelSelector:
                   matchExpressions:
                   - key: istio.io/rev
                     operator: In
                     values:
                     - main
                 topologyKey: topology.kubernetes.io/zone
               weight: 100
       enabled: true
       env:
         # Enables automatic creation of remote peer gateways
         PEERING_AUTOMATIC_LOCAL_GATEWAY: "true"
         PILOT_ENABLE_IP_AUTOALLOCATE: "true"
         PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
         PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
         AUTO_RELOAD_PLUGIN_CERTS: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS: "true"
     platforms:
       peering:
         enabled: true
     revisionTags:
     - default
     license:
       value: ${GLOO_MESH_LICENSE_KEY}
       # secretRef:
       #   name: 
       #   namespace: 
     EOF
       

   • Istio 1.24 and earlier:

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     global:
       hub: ${REPO}
       variant: distroless
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
     profile: ambient
     revision: main
     istio_cni:
       enabled: true
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       trustDomain: ${CLUSTER_NAME}
     pilot:
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
             - podAffinityTerm:
                 labelSelector:
                   matchExpressions:
                   - key: istio.io/rev
                     operator: In
                     values:
                     - main
                 topologyKey: topology.kubernetes.io/zone
               weight: 100
       enabled: true
       env:
         PILOT_ENABLE_IP_AUTOALLOCATE: "true"
         PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
         PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
         AUTO_RELOAD_PLUGIN_CERTS: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS: "true"
     platforms:
       peering:
         enabled: true
     revisionTags:
     - default
     EOF
       

   • Istio 1.25 and later: If you prefer to specify your license secret instead of an inline value, you can uncomment the secretRef fields.

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     global:
       hub: ${REPO}
       variant: distroless
       platform: openshift
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
     profile: ambient
     revision: main
     istio_cni:
       enabled: true
       namespace: kube-system
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       trustDomain: ${CLUSTER_NAME}
     pilot:
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
             - podAffinityTerm:
                 labelSelector:
                   matchExpressions:
                   - key: istio.io/rev
                     operator: In
                     values:
                     - main
                 topologyKey: topology.kubernetes.io/zone
               weight: 100
       enabled: true
       env:
         PILOT_ENABLE_IP_AUTOALLOCATE: "true"
         PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
         PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
         AUTO_RELOAD_PLUGIN_CERTS: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS: "true"
     platforms:
       peering:
         enabled: true
     revisionTags:
     - default
     license:
       value: ${GLOO_MESH_LICENSE_KEY}
       # secretRef:
       #   name: 
       #   namespace: 
     EOF
       

   • Istio 1.24 and earlier:

       helm upgrade --install istiod oci://${HELM_REPO}/istiod \
     --namespace istio-system \
     --kube-context ${CLUSTER_CONTEXT} \
     --version ${ISTIO_IMAGE} \
     -f - <<EOF
     global:
       hub: ${REPO}
       variant: distroless
       platform: openshift
       proxy:
         clusterDomain: cluster.local
       tag: ${ISTIO_IMAGE}
       multiCluster:
         clusterName: ${CLUSTER_NAME}
       network: ${CLUSTER_NAME}
     profile: ambient
     revision: main
     istio_cni:
       enabled: true
       namespace: kube-system
     meshConfig:
       accessLogFile: /dev/stdout
       defaultConfig:
         proxyMetadata:
           ISTIO_META_DNS_AUTO_ALLOCATE: "true"
           ISTIO_META_DNS_CAPTURE: "true"
       trustDomain: ${CLUSTER_NAME}
     pilot:
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
             - podAffinityTerm:
                 labelSelector:
                   matchExpressions:
                   - key: istio.io/rev
                     operator: In
                     values:
                     - main
                 topologyKey: topology.kubernetes.io/zone
               weight: 100
       enabled: true
       env:
         PILOT_ENABLE_IP_AUTOALLOCATE: "true"
         PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
         PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
         AUTO_RELOAD_PLUGIN_CERTS: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION: "true"
         PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS: "true"
     platforms:
       peering:
         enabled: true
     revisionTags:
     - default
     EOF
       

5. Install the Istio CNI node agent daemonset. Note that although the CNI is included in this section, it is technically not part of the control plane or data plane.

     helm upgrade --install istio-cni oci://${HELM_REPO}/cni \
   --namespace istio-system \
   --kube-context=${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   global:
     hub: ${REPO}
     proxy: ${ISTIO_IMAGE}
     variant: distroless
   profile: ambient
   revision: main
   cni:
     ambient:
       dnsCapture: true
     excludeNamespaces:
     - istio-system
     - kube-system
   EOF
     

     helm upgrade --install istio-cni oci://${HELM_REPO}/cni \
   --namespace kube-system \
   --kube-context=${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   --create-namespace \
   -f - <<EOF
   global:
     hub: ${REPO}
     proxy: ${ISTIO_IMAGE}
     variant: distroless
     platform: openshift
   profile: ambient
   revision: main
   cni:
     ambient:
       dnsCapture: true
     excludeNamespaces:
     - istio-system
     - kube-system
   EOF
     

6. Verify that the components of the Istio control plane are successfully installed. Because the Istio CNI is deployed as a daemon set, the number of CNI pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.

     kubectl get pods -A --context ${CLUSTER_CONTEXT} | grep istio
     

   Example output:

     istio-system   istio-cni-node-g84js                   1/1     Running   0          8s
   istio-system   istio-cni-node-gwssc                   1/1     Running   0          8s
   istio-system   istiod-main-7c59ff7bd7-4mc25           1/1     Running   0          19s
     

7. Install the ztunnel daemonset. Note that this component is required to successfully peer clusters together.

     helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
   --namespace istio-system \
   --kube-context=${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   configValidation: true
   enabled: true
   env:
     L7_ENABLED: "true"
     ISTIO_META_DNS_CAPTURE: "true"
     SKIP_VALIDATE_TRUST_DOMAIN: "true"
   hub: ${REPO}
   istioNamespace: istio-system
   multiCluster:
     clusterName: ${CLUSTER_NAME}
   network: ${CLUSTER_NAME}
   namespace: istio-system
   profile: ambient
   proxy:
     clusterDomain: cluster.local
   tag: ${ISTIO_IMAGE}
   variant: distroless
   revision: main
   EOF
     

     helm upgrade --install ztunnel oci://${HELM_REPO}/ztunnel \
   --namespace kube-system \
   --kube-context=${CLUSTER_CONTEXT} \
   --version ${ISTIO_IMAGE} \
   -f - <<EOF
   configValidation: true
   enabled: true
   env:
     L7_ENABLED: "true"
     ISTIO_META_DNS_CAPTURE: "true"
     SKIP_VALIDATE_TRUST_DOMAIN: "true"
   hub: ${REPO}
   istioNamespace: istio-system
   multiCluster:
     clusterName: ${CLUSTER_NAME}
   network: ${CLUSTER_NAME}
   namespace: kube-system
   profile: ambient
   global: 
     platform: openshift
   proxy:
     clusterDomain: cluster.local
   tag: ${ISTIO_IMAGE}
   terminationGracePeriodSeconds: 29
   variant: distroless
   revision: main
   EOF
     

8. Verify that the ztunnel pods are successfully installed. Because the ztunnel is deployed as a daemon set, the number of pods equals the number of nodes in your cluster. Note that it might take a few seconds for the pods to become available.

     kubectl get pods -A --context ${CLUSTER_CONTEXT} | grep ztunnel
     

   Example output:

     ztunnel-tvtzn             1/1     Running   0          7s
   ztunnel-vtpjm             1/1     Running   0          4s
   ztunnel-hllxg             1/1     Running   0          4s
     

9. Label the istio-system namespace with the cluster’s network name, which you previously set to your cluster’s name in the global.network field of the istiod installation. The Istio control plane uses this label internally to group pods that exist in the same L3 network.

     kubectl label namespace istio-system --context ${CLUSTER_CONTEXT} topology.istio.io/network=${CLUSTER_NAME}
     

10. Create an east-west gateway in the istio-eastwest namespace. An east-west gateway facilitates traffic between services in each cluster in your multicluster mesh.

    • You can use the following istioctl command to quickly create the east-west gateway.

        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
      istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT}
        

    • To take a look at the Gateway resource that this command creates, you can include the --generate flag in the command.

        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
      istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT} --generate
        

      In this example output, the gatewayClassName that is used, istio-eastwest, is included by default when you install Istio in ambient mode.

        apiVersion: gateway.networking.k8s.io/v1
      kind: Gateway
      metadata:
        labels:
          istio.io/expose-istiod: "15012"
          topology.istio.io/network: "<cluster_network_name>"
        name: istio-eastwest
        namespace: istio-eastwest
      spec:
        gatewayClassName: istio-eastwest
        listeners:
        - name: cross-network
          port: 15008
          protocol: HBONE
          tls:
            mode: Passthrough
        - name: xds-tls
          port: 15012
          protocol: TLS
          tls:
            mode: Passthrough
        

11. Verify that the east-west gateway is successfully deployed.

      kubectl get pods -n istio-eastwest --context $CLUSTER_CONTEXT
      

12. For each cluster that you want to include in the multicluster service mesh setup, including the management cluster, repeat these steps to deploy service mesh components and an east-west gateway in each cluster. Remember to change the cluster name and context variables each time you repeat the steps.

      export CLUSTER_NAME=<cluster-name>
    export CLUSTER_CONTEXT=<cluster-context>
      

Review remote peer gateways link

After you complete the steps for each cluster, verify that Gloo Mesh successfully created and distributed the remote peering gateways. These gateways use the istio-remote GatewayClass, which allows the istiod control plane in each cluster to discover the east-west gateway addresses of other clusters. Gloo Mesh generates one istio-remote resource in the management cluster for each connected workload cluster, and then distributes the gateway to each cluster respectively.

1. Verify that an istio-remote gateway for each connected cluster is copied to the management cluster.

     kubectl get gateways -n istio-eastwest --context $MGMT_CONTEXT
     

   In this example output, the istio-remote gateways that were auto-generated for workload clusters cluster1 and cluster2 are copied to the management cluster, alongside the management cluster’s own istio-remote gateway and east-west gateway.

     NAMESPACE        NAME                            CLASS           ADDRESS                                                                   PROGRAMMED   AGE
   istio-eastwest   istio-eastwest                 istio-eastwest   a7f6f1a2611fc4eb3864f8d688622fd4-1234567890.us-east-1.elb.amazonaws.com   True         6s
   istio-eastwest   istio-remote-peer-cluster1     istio-remote     a5082fe9522834b8192a6513eb8c6b01-0987654321.us-east-1.elb.amazonaws.com   True         4s
   istio-eastwest   istio-remote-peer-cluster2     istio-remote     aaad62dc3ffb142a1bfc13df7fe9665b-5678901234.us-east-1.elb.amazonaws.com   True         4s
   istio-eastwest   istio-remote-peer-mgmt         istio-remote     a7f6f1a2611fc4eb3864f8d688622fd4-1234567890.us-east-1.elb.amazonaws.com   True         4s
     

2. In each cluster, verify that all istio-remote gateways are successfully distributed to all workload clusters. This ensures that services in each workload cluster can now access the east-west gateways in other clusters of the multicluster mesh setup.

     kubectl get gateways -n istio-eastwest --context $CLUSTER_CONTEXT
     

Next link

Add apps to the sidecar mesh. For multicluster setups, this includes making specific services available across your linked cluster setup.