Use the Gloo Operator to link ambient service meshes across multiple clusters.

Overview

In this guide, you deploy an ambient mesh to each workload cluster, create an east-west gateway in each cluster, and link the istiod control planes across cluster networks by using peering gateways. In the next guide, you can deploy the Bookinfo sample app to the ambient mesh in each cluster, and make select services available across the multicluster mesh. Incoming requests can then be routed from an ingress gateway, such as Solo Enterprise for kgateway, to services in your mesh across all clusters.

The following diagram demonstrates an ambient mesh setup across multiple clusters. For more information about the components that are installed in these steps, see the ambient components overview.

Considerations

Before you set up a multicluster ambient mesh, review the following considerations and requirements.

License requirements

Version requirements

Multicluster setups require the Solo distribution of Istio version 1.24.3 or later (1.24.3-solo), including the Solo distribution of istioctl.

Revision and canary upgrade limitations

The upgrade guides in this documentation show you how to perform in-place upgrades for your Istio components, which is the recommended upgrade strategy.

Multicluster setup method

To get started, choose one of following methods for creating a multicluster mesh.

• Option 1: Link ambient meshes by deploying new mesh installations, or upgrading existing ones.
• Option 2: Automatically link clusters by using the Gloo management plane. Note that this method is a beta feature.

Option 1: Link ambient meshes

In each cluster, use the Gloo Operator to upgrade existing or create new ambient mesh components. Then, create east-west gateways so that traffic requests can be routed cross-cluster, and link clusters to enable cross-cluster service discovery.

Set up tools

1. Set your Enterprise level license for Solo Enterprise for Istio as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing. Note that you might have previously saved this key in another variable, such as ${SOLO_LICENSE_KEY} or ${GLOO_MESH_LICENSE_KEY}.

   export SOLO_ISTIO_LICENSE_KEY=<enterprise_license_key>

2. Choose the version of Istio that you want to install or upgrade to by reviewing the supported versions.

3. Save the Solo distribution of Istio version.

   export ISTIO_VERSION=1.27.8
   export ISTIO_IMAGE=${ISTIO_VERSION}-solo

4. Save the repo key for the minor version of the Solo distribution of Istio that you want to install. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Istio images built by Solo.io support article.

   # 12-character hash at the end of the repo URL
   export REPO_KEY=<repo_key>
   export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
   export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}

5. Get the Solo distribution of Istio binary and install istioctl, which you use for multicluster linking and gateway commands. This script automatically detects your OS and architecture, downloads the appropriate Solo distribution of Istio binary, and verifies the installation.

   bash <(curl -sSfL https://raw.githubusercontent.com/solo-io/doc-examples/main/istio/install-istioctl.sh)
   export PATH=${HOME}/.istioctl/bin:${PATH}

6. Save the names and kubeconfig contexts of each cluster. This guide uses two clusters as an example. To add more clusters to the multicluster setup, include them in the arrays.

   export cluster1=<cluster1_name>
   export context1=<cluster1_context>
   export cluster2=<cluster2_name>
   export context2=<cluster2_context>

Create a shared root of trust

Each cluster in the multicluster setup must have a shared root of trust. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}

mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

function create_cacerts_secret() {
  context=${1:?context}
  cluster=${2:?cluster}
  make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
  kubectl --context=${context} create ns istio-system || true
  kubectl --context=${context} create secret generic cacerts -n istio-system \
    --from-file=${cluster}/ca-cert.pem \
    --from-file=${cluster}/ca-key.pem \
    --from-file=${cluster}/root-cert.pem \
    --from-file=${cluster}/cert-chain.pem
}

create_cacerts_secret ${context1} ${cluster1}
create_cacerts_secret ${context2} ${cluster2}

cd ../..

Upgrade or deploy ambient components

In each cluster, use the Gloo Operator to upgrade existing or create new ambient mesh components. Then, create east-west gateways so that traffic requests can be routed cross-cluster, and link clusters to enable cross-cluster service discovery.

Upgrade ambient settings

In each cluster, upgrade your existing ambient meshes installed with the Gloo Operator for multicluster. Then, create east-west gateways so that traffic requests can be routed cross-cluster.

1. Update your managed-istio ServiceMeshController resources to include the required multicluster settings in both clusters.

   function upgrade_smc() {
     context=${1:?context}
     cluster=${2:?cluster}
     kubectl apply -n gloo-mesh --context=${context} -f - <<EOF
   apiVersion: operator.gloo.solo.io/v1
   kind: ServiceMeshController
   metadata:
     name: managed-istio
     labels:
       app.kubernetes.io/name: managed-istio
   spec:
     cluster: ${cluster}
     network: ${cluster}
     dataplaneMode: Ambient
     installNamespace: istio-system
     version: ${ISTIO_VERSION}
   EOF
   }
   
   upgrade_smc ${context1} ${cluster1}
   upgrade_smc ${context2} ${cluster2}
   info

   Note that the operator detects your cloud provider and cluster platform, and configures the necessary settings required for that platform for you. For example, if you create an ambient mesh in an OpenShift cluster, no OpenShift-specific settings are required in the ServiceMeshController, because the operator automatically sets the appropriate settings for OpenShift and your specific cloud provider accordingly.
   
   If you set the installNamespace to a namespace other than gloo-system, gloo-mesh, or istio-system, you must include the ‐‐set manager.env.WATCH_NAMESPACES=<namespace> setting.

2. Verify that the ServiceMeshController is ready in both clusters. In the Status section of the output, make sure that all statuses are True, and that the phase is SUCCEEDED.

   kubectl describe servicemeshcontroller -n gloo-mesh managed-istio --context ${context1}
   kubectl describe servicemeshcontroller -n gloo-mesh managed-istio --context ${context2}

   Example output:

   ...
   Status:
     Conditions:
       Last Transition Time:  2024-12-27T20:47:01Z
       Message:               Manifests initialized
       Observed Generation:   1
       Reason:                ManifestsInitialized
       Status:                True
       Type:                  Initialized
       Last Transition Time:  2024-12-27T20:47:02Z
       Message:               CRDs installed
       Observed Generation:   1
       Reason:                CRDInstalled
       Status:                True
       Type:                  CRDInstalled
       Last Transition Time:  2024-12-27T20:47:02Z
       Message:               Deployment succeeded
       Observed Generation:   1
       Reason:                DeploymentSucceeded
       Status:                True
       Type:                  ControlPlaneDeployed
       Last Transition Time:  2024-12-27T20:47:02Z
       Message:               Deployment succeeded
       Observed Generation:   1
       Reason:                DeploymentSucceeded
       Status:                True
       Type:                  CNIDeployed
       Last Transition Time:  2024-12-27T20:47:02Z
       Message:               Deployment succeeded
       Observed Generation:   1
       Reason:                DeploymentSucceeded
       Status:                True
       Type:                  WebhookDeployed
       Last Transition Time:  2024-12-27T20:47:02Z
       Message:               All conditions are met
       Observed Generation:   1
       Reason:                SystemReady
       Status:                True
       Type:                  Ready
     Phase:                   SUCCEEDED
   Events:                    <none>

3. Create an east-west gateway in the istio-eastwest namespace. In each cluster, the east-west gateway is implemented as a ztunnel that facilitates traffic between services across clusters in your multicluster mesh.

   Create the east-west gateway in both clusters. For customization options, see the gateway guide in the Istio docs.

   function create_ew_gateway() {
     context=${1:?context}
     cluster=${2:?cluster}
     kubectl create namespace istio-eastwest --context ${context}
     istioctl multicluster expose --namespace istio-eastwest --context ${context} --generate > ew-gateway-${cluster}.yaml
     kubectl apply -f ew-gateway-${cluster}.yaml --context ${context}
   }
   
   create_ew_gateway ${context1} ${cluster1}
   create_ew_gateway ${context2} ${cluster2}

   In this example generated Gateway resource, the gatewayClassName that is used, istio-eastwest, is included by default when you install Istio in ambient mode. For customization options, see the gateway guide in the Istio docs.

   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     labels:
       istio.io/expose-istiod: "15012"
       topology.istio.io/network: "cluster1"
       topology.kubernetes.io/region: "us-east"
       topology.kubernetes.io/zone: "us-east-1"
     name: istio-eastwest
     namespace: istio-eastwest
   spec:
     gatewayClassName: istio-eastwest
     listeners:
     - name: cross-network
       port: 15008
       protocol: HBONE
       tls:
         mode: Passthrough
     - name: xds-tls
       port: 15012
       protocol: TLS
       tls:
         mode: Passthrough

4. Verify that the east-west gateway is successfully deployed in both clusters.

   kubectl get svc -n istio-eastwest --context ${context1}
   kubectl get svc -n istio-eastwest --context ${context2}

   Example output:

   NAME             TYPE           CLUSTER-IP       EXTERNAL-IP             PORT(S)                                           AGE
   istio-eastwest   LoadBalancer   172.20.205.104   <external_address>      15021:31655/TCP,15008:32699/TCP,15012:32166/TCP   55s
   NAME             TYPE           CLUSTER-IP       EXTERNAL-IP             PORT(S)                                           AGE
   istio-eastwest   LoadBalancer   172.20.21.117    <external_address>      15021:30324/TCP,15008:31875/TCP,15012:31050/TCP   77s

Deploy ambient components

If you do not have ambient meshes yet, use the Gloo Operator to deploy ambient mesh components to each cluster. Then, create east-west gateways so that traffic requests can be routed cross-cluster, and link clusters to enable cross-cluster service discovery.

1. Install the Gloo Operator to the gloo-mesh namespace. This operator deploys and manages your Istio installation. For more information, see the Helm reference.Note that if you already installed Solo Enterprise for Istio, you can optionally reference the secret that Solo Enterprise for Istio automatically creates for your license in the –set manager.env.SOLO_ISTIO_LICENSE_KEY_SECRET_REF=gloo-mesh/license-keys flag instead.

   function install_operator() {
     context=${1:?context}
     helm install gloo-operator oci://us-docker.pkg.dev/solo-public/gloo-operator-helm/gloo-operator \
       --version 0.5.2 \
       -n gloo-mesh \
       --create-namespace \
       --kube-context ${context} \
       --set manager.env.SOLO_ISTIO_LICENSE_KEY=${SOLO_ISTIO_LICENSE_KEY}
   }
   
   install_operator ${context1}
   install_operator ${context2}

2. Verify that the operator pod is running in both clusters.

   kubectl get pods -n gloo-mesh --context ${context1} -l app.kubernetes.io/name=gloo-operator
   kubectl get pods -n gloo-mesh --context ${context2} -l app.kubernetes.io/name=gloo-operator

   Example output:

   NAME                             READY   STATUS    RESTARTS   AGE
   gloo-operator-749f8c679f-gw6h8   1/1     Running   0          32s
   NAME                             READY   STATUS    RESTARTS   AGE
   gloo-operator-749f8c679f-k66sk   1/1     Running   0          28s
3. Create a ServiceMeshController custom resource to configure an Istio installation. For a description of each configurable field, see the ServiceMeshController reference. If you need to set more advanced Istio configuration, you can also create a gloo-extensions-config configmap.

   function create_smc() {
     context=${1:?context}
     cluster=${2:?cluster}
     kubectl apply -n gloo-mesh --context ${context} -f -<<EOF
   apiVersion: operator.gloo.solo.io/v1
   kind: ServiceMeshController
   metadata:
     name: managed-istio
     labels:
       app.kubernetes.io/name: managed-istio
   spec:
     cluster: ${cluster}
     network: ${cluster}
     dataplaneMode: Ambient
     installNamespace: istio-system
     version: ${ISTIO_VERSION}
   EOF
   }
   
   create_smc ${context1} ${cluster1}
   create_smc ${context2} ${cluster2}
   info

   Note that the operator detects your cloud provider and cluster platform, and configures the necessary settings required for that platform for you. For example, if you create an ambient mesh in an OpenShift cluster, no OpenShift-specific settings are required in the ServiceMeshController, because the operator automatically sets the appropriate settings for OpenShift and your specific cloud provider accordingly.
   
   If you set the installNamespace to a namespace other than gloo-system, gloo-mesh, or istio-system, you must include the ‐‐set manager.env.WATCH_NAMESPACES=<namespace> setting.

4. Verify that the ServiceMeshController is ready in both clusters. In the Status section of the output, make sure that all statuses are True, and that the phase is SUCCEEDED.

   kubectl describe servicemeshcontroller -n gloo-mesh managed-istio --context ${context1}
   kubectl describe servicemeshcontroller -n gloo-mesh managed-istio --context ${context2}

   Example output:

   ...
   Status:
     Conditions:
       Last Transition Time:  2024-12-27T20:47:01Z
       Message:               Manifests initialized
       Observed Generation:   1
       Reason:                ManifestsInitialized
       Status:                True
       Type:                  Initialized
       Last Transition Time:  2024-12-27T20:47:02Z
       Message:               CRDs installed
       Observed Generation:   1
       Reason:                CRDInstalled
       Status:                True
       Type:                  CRDInstalled
       Last Transition Time:  2024-12-27T20:47:02Z
       Message:               Deployment succeeded
       Observed Generation:   1
       Reason:                DeploymentSucceeded
       Status:                True
       Type:                  ControlPlaneDeployed
       Last Transition Time:  2024-12-27T20:47:02Z
       Message:               Deployment succeeded
       Observed Generation:   1
       Reason:                DeploymentSucceeded
       Status:                True
       Type:                  CNIDeployed
       Last Transition Time:  2024-12-27T20:47:02Z
       Message:               Deployment succeeded
       Observed Generation:   1
       Reason:                DeploymentSucceeded
       Status:                True
       Type:                  WebhookDeployed
       Last Transition Time:  2024-12-27T20:47:02Z
       Message:               All conditions are met
       Observed Generation:   1
       Reason:                SystemReady
       Status:                True
       Type:                  Ready
     Phase:                   SUCCEEDED
   Events:                    <none>

5. Verify that the istiod control plane, Istio CNI, and ztunnel pods are running in both clusters.

   kubectl get pods -n istio-system --context ${context1}
   kubectl get pods -n istio-system --context ${context2}

   Example output:

   NAME                           READY   STATUS    RESTARTS   AGE
   istio-cni-node-bghxw           1/1     Running   0          26s
   istio-cni-node-n4d6v           1/1     Running   0          26s
   istiod-gloo-546f6478fb-gxx57   1/1     Running   0          18s
   istiod-gloo-546f6478fb-n7782   1/1     Running   0          33s
   ztunnel-lfzh9                  1/1     Running   0          25s
   ztunnel-rmxw6                  1/1     Running   0          25s
   NAME                           READY   STATUS    RESTARTS   AGE
   istio-cni-node-2pg9n           1/1     Running   0          26s
   istio-cni-node-mzwb6           1/1     Running   0          26s
   istiod-gloo-57f5dd4c45-2xjbk   1/1     Running   0          18s
   istiod-gloo-57f5dd4c45-wvvqz   1/1     Running   0          33s
   ztunnel-574hc                  1/1     Running   0          25s
   ztunnel-v6blv                  1/1     Running   0          25s

6. Apply the CRDs for the Kubernetes Gateway API to your cluster, which are required to create components such as waypoint proxies for L7 traffic policies, gateways with the Gateway resource, and more.

   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml --context ${context1}
   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml --context ${context2}

7. Create an east-west gateway in the istio-eastwest namespace. In each cluster, the east-west gateway is implemented as a ztunnel that facilitates traffic between services across clusters in your multicluster mesh.

   Create the east-west gateway in both clusters. For customization options, see the gateway guide in the Istio docs.

   function create_ew_gateway() {
     context=${1:?context}
     cluster=${2:?cluster}
     kubectl create namespace istio-eastwest --context ${context}
     istioctl multicluster expose --namespace istio-eastwest --context ${context} --generate > ew-gateway-${cluster}.yaml
     kubectl apply -f ew-gateway-${cluster}.yaml --context ${context}
   }
   
   create_ew_gateway ${context1} ${cluster1}
   create_ew_gateway ${context2} ${cluster2}

   In this example generated Gateway resource, the gatewayClassName that is used, istio-eastwest, is included by default when you install Istio in ambient mode. For customization options, see the gateway guide in the Istio docs.

   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     labels:
       istio.io/expose-istiod: "15012"
       topology.istio.io/network: "cluster1"
       topology.kubernetes.io/region: "us-east"
       topology.kubernetes.io/zone: "us-east-1"
     name: istio-eastwest
     namespace: istio-eastwest
   spec:
     gatewayClassName: istio-eastwest
     listeners:
     - name: cross-network
       port: 15008
       protocol: HBONE
       tls:
         mode: Passthrough
     - name: xds-tls
       port: 15012
       protocol: TLS
       tls:
         mode: Passthrough

8. Verify that the east-west gateway is successfully deployed in both clusters.

   kubectl get svc -n istio-eastwest --context ${context1}
   kubectl get svc -n istio-eastwest --context ${context2}

   Example output:

   NAME             TYPE           CLUSTER-IP       EXTERNAL-IP             PORT(S)                                           AGE
   istio-eastwest   LoadBalancer   172.20.205.104   <external_address>      15021:31655/TCP,15008:32699/TCP,15012:32166/TCP   55s
   NAME             TYPE           CLUSTER-IP       EXTERNAL-IP             PORT(S)                                           AGE
   istio-eastwest   LoadBalancer   172.20.21.117    <external_address>      15021:30324/TCP,15008:31875/TCP,15012:31050/TCP   77s

Link clusters

Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters.

1. Optional: Before you link clusters, you can check the individual readiness of each cluster for linking by running the istioctl multicluster check --precheck command. For more information about this command, see the CLI reference. If any checks fail, run the command with --verbose, and see Validate your multicluster setup.

   istioctl multicluster check --precheck --contexts="$context1,$context2"

   Before continuing to the next step, make sure that the following checks pass as expected:
   ✅ Relevant environment variables on istiod are supported.
   ✅ The license in use by istiod supports multicluster.
   ✅ All istiod, ztunnel, and east-west gateway pods are healthy.
   ✅ The east-west gateway is programmed.

2. Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters. Note that you can either link the clusters bi-directionally or asymmetrically. In a standard bi-directional setup, services in any of the linked clusters can send requests to and receive requests from the services in any of the other linked clusters. In an asymmetrical setup, you allow one cluster to send requests to another cluster, but the other cluster cannot send requests back to the first cluster.

   In the Solo distribution of Istio 1.29 or later, you can use the peering Helm chart to link your clusters.

   1. Get the addresses of the east-west gateway in each cluster. The following commands show examples for two clusters.

      export CLUSTER1_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context1} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
      export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
      echo "Cluster1 east-west gateway: $CLUSTER1_EW_ADDRESS"
      echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"

   2. Link the clusters by creating Helm releases in each cluster to represent the other clusters. In each cluster where you create Helm releases, a Gateway resource is created that uses the istio-remote GatewayClass. This class allows the gateway to connect to other clusters by using the addresses of the east-west gateways.

      • Be sure to update the region, and optionally zone, of each cluster.
      • If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, change the preferredDataplaneServiceType to nodeport. This automatically configures the peering gateways for NodePort-based cross-cluster traffic. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.
      • The following example depicts a bi-directional setup. In an asymmetrical setup, you create Helm releases to only represent the directionality you want to allow.

      helm upgrade -i peering-remote oci://${HELM_REPO}/peering \
        --version ${ISTIO_IMAGE} \
        --namespace istio-eastwest \
        --kube-context ${context1} \
        -f - <<EOF
      remote:
        create: true
        items:
          # Remote peer configuration for cluster2
          - name: istio-remote-peer-${cluster2}
            cluster: ${cluster2}
            network: ${cluster2}
            # Change to "Hostname" if using hostname
            addressType: IPAddress
            address: ${CLUSTER2_EW_ADDRESS}
            # Change to "nodeport" if using NodePorts
            preferredDataplaneServiceType: loadbalancer
            trustDomain: cluster.local
            region: "<cluster2_region>"
            # Uncomment as needed
            # zone: "<cluster2_zone>"
      EOF
      helm upgrade -i peering-remote oci://${HELM_REPO}/peering \
        --version ${ISTIO_IMAGE} \
        --namespace istio-eastwest \
        --kube-context ${context2} \
        -f - <<EOF
      remote:
        create: true
        items:
          # Remote peer configuration for cluster1
          - name: istio-remote-peer-${cluster1}
            cluster: ${cluster1}
            network: ${cluster1}
            # Change to "Hostname" if using hostname
            addressType: IPAddress
            address: ${CLUSTER1_EW_ADDRESS}
            # Change to "nodeport" if using NodePorts
            preferredDataplaneServiceType: loadbalancer
            trustDomain: cluster.local
            region: "<cluster1_region>"
            # Uncomment as needed
            # zone: "<cluster1_zone>"
      EOF

   Use the istioctl multicluster link command to quickly link clusters.

   1. Verify that the contexts for the clusters that you want to include in the multicluster mesh are listed in your kubeconfig file, which is required for the istioctl multicluster link command. If you do not have access to the kubeconfig files, use the declarative resources tab.

      kubectl config get-contexts
      • In the output, note the names of the cluster contexts, which you use in the next step to link the clusters.
      • If you have multiple kubeconfig files, you can generate a merged kubeconfig file by running the following command.

        KUBECONFIG=<kubeconfig_file1>.yaml:<file2>.yaml kubectl config view --flatten

   2. Using the names of the cluster contexts, link the clusters so that they can communicate. To take a look at the Gateway resources that this command creates, you can include the --generate flag in the command. For more information about this command, see the CLI reference.

      • Bi-directional: You can use the following istioctl command to quickly link the clusters bi-directionally. In each cluster, Gateway resources are created that use the istio-remote GatewayClass. This class allows the gateways to connect to other clusters by using the addresses of the east-west gateways.

        istioctl multicluster link --namespace istio-eastwest --contexts="$context1,$context2"

        Example output for two clusters:

        Gateway istio-eastwest/istio-remote-peer-cluster1 applied to cluster "cluster2" pointing to cluster "cluster1" (network "cluster1")
        Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "cluster1" pointing to cluster "cluster2" (network "cluster2")

      • Asymmetrical: You can use the following istioctl command to quickly link the clusters asymmetrically. The services in the cluster in the --from flag can send requests to services in the cluster in the --to flag, but sending requests in the reverse direction is not permitted.

        For example, this command allows services in cluster1’s mesh to send requests to services in cluster2’s mesh through cluster2’s east-west gateway. However, the reverse is not permitted: services in cluster2’s mesh cannot send requests through cluster1’s east-west gateway to services in cluster1.

        istioctl multicluster link --namespace istio-eastwest --from ${context1} --to ${context2}

        Example output:

        Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "cluster1" pointing to cluster "cluster2" (network "cluster2")

   Link the clusters by declaratively creating istio-remote peer gateways.

   Bi-directional: Use the following Gateway resources to create an istio-remote peer gateway in each cluster. The istio-remote GatewayClass allows the gateways to connect to other clusters by using the addresses of the east-west gateways.

   1. Get the addresses of the east-west gateway in each cluster. The following commands show examples for two clusters.

      export CLUSTER1_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context1} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
      export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
      
      echo "Cluster1 east-west gateway: $CLUSTER1_EW_ADDRESS"
      echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"

   2. Using the east-west gateway addresses, create a Gateway resource in each cluster to represent the other cluster.

      • In the labels section, be sure to update the locality labels according to the region, and optionally zone, of each cluster. For more information about locality support, see the release notes.

      kubectl apply --context ${context1} -f- <<EOF
      apiVersion: gateway.networking.k8s.io/v1
      kind: Gateway
      metadata:
        annotations:
          gateway.istio.io/service-account: istio-eastwest
          gateway.istio.io/trust-domain: ${cluster2}
        labels:
          topology.istio.io/network: ${cluster2}
          topology.kubernetes.io/region: "<cluster2_region>"
          # Uncomment as needed
          # topology.kubernetes.io/zone: "<cluster2_zone>"
        name: istio-remote-peer-${cluster2}
        namespace: istio-eastwest
      spec:
        addresses:
        # Change to 'type: Hostname' as needed, such as for AWS hostnames
        - type: IPAddress
          value: $CLUSTER2_EW_ADDRESS
        gatewayClassName: istio-remote
        listeners:
        - name: cross-network
          port: 15008
          protocol: HBONE
          tls:
            mode: Passthrough
        - name: xds-tls
          port: 15012
          protocol: TLS
          tls:
            mode: Passthrough
      EOF
      kubectl apply --context ${context2} -f- <<EOF
      apiVersion: gateway.networking.k8s.io/v1
      kind: Gateway
      metadata:
        annotations:
          gateway.istio.io/service-account: istio-eastwest
          gateway.istio.io/trust-domain: ${cluster1}
        labels:
          topology.istio.io/network: ${cluster1}
          topology.kubernetes.io/region: "<cluster1_region>"
          # Uncomment as needed
          # topology.kubernetes.io/zone: "<cluster1_zone>"
        name: istio-remote-peer-${cluster1}
        namespace: istio-eastwest
      spec:
        addresses:
        # Change to 'type: Hostname' as needed, such as for AWS hostnames
        - type: IPAddress
          value: $CLUSTER1_EW_ADDRESS
        gatewayClassName: istio-remote
        listeners:
        - name: cross-network
          port: 15008
          protocol: HBONE
          tls:
            mode: Passthrough
        - name: xds-tls
          port: 15012
          protocol: TLS
          tls:
            mode: Passthrough
      EOF

   Asymmetrical: Use the following Gateway resources to create an istio-remote peer gateway in only some clusters. The istio-remote GatewayClass allows the gateway in one cluster to connect to another cluster by using the address of the east-west gateway, but sending requests in the reverse direction is not permitted.

   1. Get the address of the east-west gateway in the cluster that you want to send traffic to. The following command shows an example for cluster2.

      export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context ${context2} -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
      
      echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"

   2. Using the east-west gateway address, create a Gateway resource in the cluster that you want to send requests from. For example, this Gateway resource allows services in cluster1’s mesh to send requests to services in cluster2’s mesh through cluster2’s east-west gateway. However, the reverse is not permitted: services in cluster2’s mesh cannot send requests through cluster1’s east-west gateway to services in cluster1.

      • In the labels section, be sure to update the locality labels according to the region, and optionally zone, of each cluster. For more information about locality support, see the release notes.

      kubectl apply --context ${context1} -f- <<EOF
      apiVersion: gateway.networking.k8s.io/v1
      kind: Gateway
      metadata:
        annotations:
          gateway.istio.io/service-account: istio-eastwest
          gateway.istio.io/trust-domain: ${cluster2}
        labels:
          topology.istio.io/network: ${cluster2}
          topology.kubernetes.io/region: "<cluster2_region>"
          # Uncomment as needed
          # topology.kubernetes.io/zone: "<cluster2_zone>"
        name: istio-remote-peer-${cluster2}
        namespace: istio-eastwest
      spec:
        addresses:
        # Change to 'type: Hostname' as needed, such as for AWS hostnames
        - type: IPAddress
          value: $CLUSTER2_EW_ADDRESS
        gatewayClassName: istio-remote
        listeners:
        - name: cross-network
          port: 15008
          protocol: HBONE
          tls:
            mode: Passthrough
        - name: xds-tls
          port: 15012
          protocol: TLS
          tls:
            mode: Passthrough
      EOF

3. Verify that peer linking was successful by running the istioctl multicluster check command. If any checks fail, run the command with --verbose, and see Validate your multicluster setup.

   istioctl multicluster check --contexts="$context1,$context2"

   In this example output, the remote peer gateways are successfully connected, and all other checks passed successfully. No global services exist because no app services are exposed across clusters in the multicluster mesh yet.

   === Cluster: cluster1 ===
   ✅ Incompatible Environment Variable Check: all relevant environment variables are valid
   ✅ License Check: license is valid for multicluster
   ✅ CNI DNS Capture Check: AMBIENT_DNS_CAPTURE is enabled
   ✅ Pod Check (istiod): all pods healthy
   ✅ Pod Check (ztunnel): all pods healthy
   ✅ Pod Check (eastwest gateway istio-eastwest/istio-eastwest): all pods healthy
   ✅ Gateway Check: all eastwest gateways programmed
        ✅ istio-eastwest/istio-eastwest available at aab8471c7fcfa4a3c82f2d217b015d97-396238517.us-east-1.elb.amazonaws.com
   ✅ Peers Check: all clusters connected
        ✅ Connected to gloo-gateway-docs-mgt via ab46aa29a49914da789a6d5422aca279-541415195.us-east-2.elb.amazonaws.com
   ℹ️  Shared Services Check: no globally shared services found
   ====== 
   
   === Cluster: cluster2 ===
   ✅ Incompatible Environment Variable Check: all relevant environment variables are valid
   ✅ License Check: license is valid for multicluster
   ✅ CNI DNS Capture Check: AMBIENT_DNS_CAPTURE is enabled
   ✅ Pod Check (istiod): all pods healthy
   ✅ Pod Check (ztunnel): all pods healthy
   ✅ Pod Check (eastwest gateway istio-eastwest/istio-eastwest): all pods healthy
   ✅ Gateway Check: all eastwest gateways programmed
        ✅ istio-eastwest/istio-eastwest available at ab46aa29a49914da789a6d5422aca279-541415195.us-east-2.elb.amazonaws.com
   ✅ Peers Check: all clusters connected
        ✅ Connected to gloo-mesh-core-docs-mgt via aab8471c7fcfa4a3c82f2d217b015d97-396238517.us-east-1.elb.amazonaws.com
   ℹ️  Shared Services Check: no globally shared services found
   ====== 
   
   ✅ Intermediate Certs Compatibility Check: all clusters have compatible intermediate certificates
   ✅ Network Configuration Check: all network configurations are valid
   ✅ Stale Workloads Check: skipped (flat network not detected)

4. Optional: Verify that the istiod control plane for each peered cluster is included in each cluster’s proxy status list.

   istioctl proxy-status --context ${context1}
   istioctl proxy-status --context ${context2}

   Example output for cluster1, in which you can verify that the istiod control plane for cluster2 is listed:

   NAME                                               CLUSTER          ISTIOD                      VERSION              SUBSCRIBED TYPES
   istio-eastwest-67fd5679dc-fhsxs.istio-eastwest     cluster1         istiod-7b7c9cc4c6-bdm9c     1.27.8-solo-fips     2 (WADS,WDS)
   istiod-6bc6765484-5bbhd.istio-system               cluster2         istiod-7b7c9cc4c6-bdm9c     1.27.8-solo-fips     3 (FSDS,SGDS,WDS)
   ztunnel-5f8rb.kube-system                          cluster1         istiod-7b7c9cc4c6-bdm9c     1.27.8-solo-fips     2 (WADS,WDS)
   ztunnel-f96kh.kube-system                          cluster1         istiod-7b7c9cc4c6-bdm9c     1.27.8-solo-fips     2 (WADS,WDS)
   ztunnel-vtj4f.kube-system                          cluster1         istiod-7b7c9cc4c6-bdm9c     1.27.8-solo-fips     2 (WADS,WDS)

Next: Add apps to the ambient mesh. For multicluster setups, this includes making specific services available across your linked cluster setup.

Option 2: Automatically link clusters (beta)

In each cluster, use the Gloo Operator to create the ambient mesh components, and create east-west gateways so that traffic requests can be routed cross-cluster. Then, use the Gloo management plane to automate multicluster linking, which enables cross-cluster service discovery.

Set up tools

1. Set your Enterprise level license for Solo Enterprise for Istio as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing. Note that you might have previously saved this key in another variable, such as ${SOLO_LICENSE_KEY} or ${GLOO_MESH_LICENSE_KEY}.

   export SOLO_ISTIO_LICENSE_KEY=<enterprise_license_key>

2. Choose the version of Istio that you want to install or upgrade to by reviewing the supported versions.

3. Save the Solo distribution of Istio version.

   export ISTIO_VERSION=1.27.8
   export ISTIO_IMAGE=${ISTIO_VERSION}-solo

4. Save the repo key for the minor version of the Solo distribution of Istio that you want to install. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Istio images built by Solo.io support article.

   # 12-character hash at the end of the repo URL
   export REPO_KEY=<repo_key>
   export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
   export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}

5. Get the Solo distribution of Istio binary and install istioctl, which you use for multicluster linking and gateway commands. This script automatically detects your OS and architecture, downloads the appropriate Solo distribution of Istio binary, and verifies the installation.

   bash <(curl -sSfL https://raw.githubusercontent.com/solo-io/doc-examples/main/istio/install-istioctl.sh)
   export PATH=${HOME}/.istioctl/bin:${PATH}

6. Save the names and kubeconfig contexts of each cluster. This guide uses two clusters as an example. To add more clusters to the multicluster setup, include them in the arrays.

   export cluster1=<cluster1_name>
   export context1=<cluster1_context>
   export cluster2=<cluster2_name>
   export context2=<cluster2_context>

Enable automatic peering of clusters

Upgrade Solo Enterprise for Istio in your multicluster setup to enable the ConfigDistribution feature flag and install the enterprise CRDs, which are required for Solo Enterprise for Istio to automate peering and distribute gateways between clusters.

1. In the cluster where you installed the management plane, upgrade your gloo-platform-crds Helm release to include the following settings.

   helm get values gloo-platform-crds -n gloo-mesh -o yaml --kube-context ${context1} > mgmt-crds.yaml
   helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \
     --kube-context ${context1} \
     --namespace gloo-mesh \
     -f mgmt-crds.yaml \
     --set featureGates.ConfigDistribution=true \
     --set installEnterpriseCrds=true

2. In the cluster where you installed the management plane, upgrade your gloo-platform Helm release to include the following settings.

   helm get values gloo-platform -n gloo-mesh -o yaml --kube-context ${context1} > mgmt-plane.yaml
   helm upgrade gloo-platform gloo-platform/gloo-platform \
     --kube-context ${context1} \
     --namespace gloo-mesh \
     -f mgmt-plane.yaml \
     --set featureGates.ConfigDistribution=true

3. In each connected cluster, upgrade your gloo-platform-crds Helm release to include the following settings. Repeat this step for each workload cluster.

   helm get values gloo-platform-crds -n gloo-mesh -o yaml --kube-context ${context2} > crds.yaml
   helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \
     --kube-context ${context2} \
     --namespace gloo-mesh \
     -f crds.yaml \
     --set installEnterpriseCrds=true

Create a shared root of trust

Each cluster in the multicluster setup must have a shared root of trust. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}

mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

function create_cacerts_secret() {
  context=${1:?context}
  cluster=${2:?cluster}
  make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
  kubectl --context=${context} create ns istio-system || true
  kubectl --context=${context} create secret generic cacerts -n istio-system \
    --from-file=${cluster}/ca-cert.pem \
    --from-file=${cluster}/ca-key.pem \
    --from-file=${cluster}/root-cert.pem \
    --from-file=${cluster}/cert-chain.pem
}

create_cacerts_secret ${context1} ${cluster1}
create_cacerts_secret ${context2} ${cluster2}

cd ../..

Deploy ambient components

1. Apply the CRDs for the Kubernetes Gateway API to your cluster, which are required to create components such as waypoint proxies for L7 traffic policies, gateways with the Gateway resource, and more.

   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml --context ${context1}
   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml --context ${context2}

2. Install the Gloo Operator to the gloo-mesh namespace. This operator deploys and manages your Istio installation. For more information, see the Helm reference.Note that if you already installed Solo Enterprise for Istio, you can optionally reference the secret that Solo Enterprise for Istio automatically creates for your license in the –set manager.env.SOLO_ISTIO_LICENSE_KEY_SECRET_REF=gloo-mesh/license-keys flag instead.

   function install_operator() {
     context=${1:?context}
     helm install gloo-operator oci://us-docker.pkg.dev/solo-public/gloo-operator-helm/gloo-operator \
       --version 0.5.2 \
       -n gloo-mesh \
       --create-namespace \
       --kube-context ${context} \
       --set manager.env.SOLO_ISTIO_LICENSE_KEY=${SOLO_ISTIO_LICENSE_KEY}
   }
   
   install_operator ${context1}
   install_operator ${context2}

3. Verify that the operator pod is running in both clusters.

   kubectl get pods -n gloo-mesh --context ${context1} -l app.kubernetes.io/name=gloo-operator
   kubectl get pods -n gloo-mesh --context ${context2} -l app.kubernetes.io/name=gloo-operator

   Example output:

   gloo-operator-78d58d5c7b-lzbr5     1/1     Running   0          48s
   gloo-operator-77h2k6sb4i-yd5wn     1/1     Running   0          49s

4. Apply the following configmap and ServiceMeshController for the Gloo Operator to enable multicluster peering and deploy an ambient mesh in both clusters.

   function create_auto_peering_smc() {
     context=${1:?context}
     cluster=${2:?cluster}
     kubectl apply -n gloo-mesh --context ${context} -f -<<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: gloo-extensions-config
     namespace: gloo-mesh
   data:
     beta: |
       serviceMeshController:
         multiClusterMode: Peering
     values.istiod: |
       env:
         PEERING_AUTOMATIC_LOCAL_GATEWAY: "true"
   EOF
     kubectl apply -n gloo-mesh --context ${context} -f -<<EOF
   apiVersion: operator.gloo.solo.io/v1
   kind: ServiceMeshController
   metadata:
     name: managed-istio
     labels:
       app.kubernetes.io/name: managed-istio
   spec:
     cluster: ${cluster}
     network: ${cluster}
     dataplaneMode: Ambient
     installNamespace: istio-system
     version: ${ISTIO_VERSION}
   EOF
   }
   
   create_auto_peering_smc ${context1} ${cluster1}
   create_auto_peering_smc ${context2} ${cluster2}