Overview

In this guide, you deploy an ambient mesh to each workload cluster, create an east-west gateway in each cluster, and link the istiod control planes across cluster networks by using peering gateways. In the next guide, you can deploy the Bookinfo sample app to the ambient mesh in each cluster, and make select services available across the multicluster mesh. Incoming requests can then be routed from an ingress gateway, such as Solo Enterprise for kgateway, to services in your mesh across all clusters.

The following diagram demonstrates an ambient mesh setup across multiple clusters.

Figure: Multicluster ambient mesh set up with the Solo distribution of Istio and Solo Enterprise for kgateway.
Figure: Multicluster ambient mesh set up with the Solo distribution of Istio and Solo Enterprise for kgateway.

For more information about in-mesh routing, check out Control in-mesh traffic with east-west gateways and waypoints. For more information about the components that are installed in these steps, see the ambient components overview.

Considerations

Before you set up a multicluster ambient mesh, review the following considerations and requirements.

License requirements

Version requirements

Multicluster setups require the Solo distribution of Istio version 1.24.3 or later (1.24.3-solo), including the Solo distribution of istioctl.

Revision and canary upgrade limitations

The upgrade guides in this documentation show you how to perform in-place upgrades for your Istio components, which is the recommended upgrade strategy.

Cross-cluster traffic addresses

In each cluster, you create an east-west gateway, which is implemented as a ztunnel that facilitates traffic between services across clusters in your multicluster mesh. In the Solo distribution of Istio 1.28 and later, you can use either LoadBalancer or NodePort addresses to resolve cross-cluster traffic requests through this gateway. Note that the NodePort method is considered beta in the Solo distribution of Istio version 1.29.

LoadBalancer: In the standard LoadBalancer peering method, cross-cluster traffic through the east-west gateway resolves to its LoadBalancer address.

NodePort (beta): If you prefer to use direct pod-to-pod traffic across clusters, you can annotate the east-west and peering gateways so that cross-cluster traffic resolves to NodePort addresses. This method allows you to avoid LoadBalancer services to reduce cross-cluster traffic costs. Review the following considerations:

  • Note that the gateways must still be created with stable IP addresses, which are required for xDS communication with the istiod control plane in each cluster. NodePort peering is used for data-plane communication, in that requests to services resolve to the NodePort instead of the LoadBalancer address. Also, the east-west gateway must have the topology.istio.io/cluster label.
  • If a node in a target cluster becomes inaccessible, such as during a restart or replacement, a delay can occur in the connection from the client cluster that must become aware of the new east-west gateway NodePort. In this case, you might see a connection error when trying to send cross-cluster traffic to an east-west gateway that is no longer accepting connections.
  • Only nodes where an east-west gateway pod is provisioned are considered targets for traffic.
  • NodePort peering uses only InternalIP node addresses. Ensure that your environment is configured so that nodes are reachable via their InternalIP address. If you need to use another address type, such as ExternalIP, contact Solo for engineering design and implementation support.
  • Like LoadBalancer gateways, NodePort gateways support traffic from Envoy-based ingress gateways, waypoints, and sidecars.
  • This feature is in a beta state. For more information, see Solo feature maturity.
  • This feature is only supported for default network setups, and is not applicable in flat network setups.

The steps in the following guide include options for either the LoadBalancer or NodePort method. A status condition on each east-west and remote peer gateway indicates which dataplane service type is in use.

For more information about nodeport peering, see the Best practices for multicluster peering.

Migrating from multicluster community Istio

If you previously used the multicluster feature in community Istio, and want to now migrate to multicluster peering in the Solo distribution of Istio, the DISABLE_LEGACY_MULTICLUSTER environment variable is introduced in the Solo distribution of Istio version 1.28 to disable the community multicluster mechanisms. Multicluster in community Istio uses remote secrets that contain kubeconfigs to watch resources on remote clusters. This system is incompatible with the decentralized, push-based model for peering in the Solo distribution of Istio. This variable causes istiod to ignore remote secrets so that it does not attempt to set up Kubernetes clients to connect to them.

  • For fresh multicluster mesh installations with the Solo distribution of Istio, use this environment variable in your istiod settings. This setting serves as a recommended safety measure to prevent any use of remote secrets.
  • If you want to initiate a multicluster migration from community Istio, contact a Solo account representative. An account representative can help you set up two revisions of Istio that each select a different set of namespaces, and set the DISABLE_LEGACY_MULTICLUSTER variable on the revision that uses the Solo distribution of Istio for multicluster peering.

Multicluster setup method

To get started, choose one of following methods for creating a multicluster mesh.

In each cluster, use the Gloo Operator to upgrade existing or create new ambient mesh components. Then, create east-west gateways so that traffic requests can be routed cross-cluster, and link clusters to enable cross-cluster service discovery.

Set up tools

  1. Set your Enterprise level license for Solo Enterprise for Istio as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing. Note that you might have previously saved this key in another variable, such as ${SOLO_LICENSE_KEY} or ${GLOO_MESH_LICENSE_KEY}.

      export SOLO_ISTIO_LICENSE_KEY=<enterprise_license_key>

  2. Choose the version of Istio that you want to install or upgrade to by reviewing the supported versions.

  3. Save the Solo distribution of Istio version.

      export ISTIO_VERSION=1.28.1
export ISTIO_IMAGE=${ISTIO_VERSION}-solo

  4. Save the image and Helm repository information for the Solo distribution of Istio.

    • Istio 1.29 and later:

        export REPO=us-docker.pkg.dev/soloio-img/istio
export HELM_REPO=us-docker.pkg.dev/soloio-img/istio-helm

    • Istio 1.28 and earlier: You must provide a repo key for the minor version of the Solo distribution of Istio that you want to install. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Istio images built by Solo.io support article.

        # 12-character hash at the end of the repo URL
export REPO_KEY=<repo_key>
export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}

  5. Get the Solo distribution of Istio binary and install istioctl, which you use for multicluster linking and gateway commands. This script automatically detects your OS and architecture, downloads the appropriate Solo distribution of Istio binary, and verifies the installation.

      bash <(curl -sSfL https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/gloo-mesh/install-istioctl.sh)
export PATH=${HOME}/.istioctl/bin:${PATH}

Create a shared root of trust

Each cluster in the multicluster setup must have a shared root of trust. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.

By default, the Istio CA generates a self-signed root certificate and key, and uses them to sign the workload certificates. For more information, see the Plug in CA Certificates guide in the community Istio documentation.

For demo installations, you can run the following function to quickly generate and plug in the certificates and key for the Istio CA:

  curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}

mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

function create_cacerts_secret() {
  context=${1:?context}
  cluster=${2:?cluster}
  make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
  kubectl --context=${context} create ns istio-system || true
  kubectl --context=${context} create secret generic cacerts -n istio-system \
    --from-file=${cluster}/ca-cert.pem \
    --from-file=${cluster}/ca-key.pem \
    --from-file=${cluster}/root-cert.pem \
    --from-file=${cluster}/cert-chain.pem
}

create_cacerts_secret ${REMOTE_CONTEXT1} ${REMOTE_CLUSTER1}
create_cacerts_secret ${REMOTE_CONTEXT2} ${REMOTE_CLUSTER2}

cd ../..

To enhance the security of your setup even further and have full control over the Istio CA lifecycle, you can generate and store the root and intermediate CA certificates and keys with your own PKI provider. You can then use tools such as cert-manager to send certificate signing requests on behalf of istiod to your PKI provider. Cert-manager stores the signed intermediate certificates and keys in the cacerts Kubernetes secret so that istiod can use these credentials to issue leaf certificates for the workloads in the service mesh. You can set up cert-manager to also check the certificates and renew them before they expire.

AWS Private CA issuer and cert-manager: For an architectural overview of this certificate setup, see Bring your own Istio CAs with AWS. For steps on how to deploy this certificate setup, check out this Solo.io blog post. Be sure to repeat the steps so that a cacerts secret exists in each cluster.

Upgrade or deploy ambient components

In each cluster, use the Gloo Operator to upgrade existing or create new ambient mesh components. Then, create east-west gateways so that traffic requests can be routed cross-cluster, and link clusters to enable cross-cluster service discovery.

Upgrade ambient settings

In each cluster, upgrade your existing ambient meshes installed with the Gloo Operator for multicluster. Then, create east-west gateways so that traffic requests can be routed cross-cluster.

  1. Save the name and kubeconfig context of a cluster where you want to install Istio in the following environment variables. Each time you repeat the steps in this guide, you change these variables to the next workload cluster’s name and context.

      export CLUSTER_NAME=<cluster-name>
export CLUSTER_CONTEXT=<cluster-context>

  2. Update your managed-istio ServiceMeshController resources to include the required multicluster settings.

      kubectl apply -n gloo-mesh --context ${CLUSTER_CONTEXT} -f - <<EOF
apiVersion: operator.gloo.solo.io/v1
kind: ServiceMeshController
metadata:
  name: managed-istio
  labels:
    app.kubernetes.io/name: managed-istio
spec:
  cluster: ${CLUSTER_NAME}
  network: ${CLUSTER_NAME}
  dataplaneMode: Ambient
  installNamespace: istio-system
  version: ${ISTIO_VERSION}
EOF

  3. Verify that the ServiceMeshController is ready. In the Status section of the output, make sure that all statuses are True, and that the phase is SUCCEEDED.

      kubectl describe servicemeshcontroller -n gloo-mesh managed-istio --context ${CLUSTER_CONTEXT}

    Example output:

      ...
Status:
  Conditions:
    Last Transition Time:  2024-12-27T20:47:01Z
    Message:               Manifests initialized
    Observed Generation:   1
    Reason:                ManifestsInitialized
    Status:                True
    Type:                  Initialized
    Last Transition Time:  2024-12-27T20:47:02Z
    Message:               CRDs installed
    Observed Generation:   1
    Reason:                CRDInstalled
    Status:                True
    Type:                  CRDInstalled
    Last Transition Time:  2024-12-27T20:47:02Z
    Message:               Deployment succeeded
    Observed Generation:   1
    Reason:                DeploymentSucceeded
    Status:                True
    Type:                  ControlPlaneDeployed
    Last Transition Time:  2024-12-27T20:47:02Z
    Message:               Deployment succeeded
    Observed Generation:   1
    Reason:                DeploymentSucceeded
    Status:                True
    Type:                  CNIDeployed
    Last Transition Time:  2024-12-27T20:47:02Z
    Message:               Deployment succeeded
    Observed Generation:   1
    Reason:                DeploymentSucceeded
    Status:                True
    Type:                  WebhookDeployed
    Last Transition Time:  2024-12-27T20:47:02Z
    Message:               All conditions are met
    Observed Generation:   1
    Reason:                SystemReady
    Status:                True
    Type:                  Ready
  Phase:                   SUCCEEDED
Events:                    <none>

  4. Create an east-west gateway in the istio-eastwest namespace. In each cluster, the east-west gateway is implemented as a ztunnel that facilitates traffic between services across clusters in your multicluster mesh. You can use either LoadBalancer or NodePort addresses for cross-cluster traffic.

    1. Create the istio-eastwest namespace.
        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
    2. Create the east-west gateway. Cross-cluster traffic though this gateway resolves to the LoadBalancer address. For customization options, see the gateway guide in the Istio docs.
      • Solo distribution of istioctl: For more information about this command, see the CLI reference.
          istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT} --generate > ew-gateway.yaml
kubectl apply -f ew-gateway.yaml --context ${CLUSTER_CONTEXT}
      • Helm: For more information about the peering chart, see the Helm values reference. For recommendations on customizing the east-west gateway for resiliency and availability with the Helm chart, see the best practices for multicluster peering.
          helm upgrade -i peering-eastwest oci://${HELM_REPO}/peering \
  --version ${ISTIO_IMAGE} \
  --namespace istio-eastwest \
  --kube-context ${CLUSTER_CONTEXT} \
  -f - <<EOF
eastwest:
  create: true
  cluster: ${CLUSTER_NAME}
  # The network that the istio-system namespace is labeled with.
  # In prod environments, network and cluster are likely not the same value.
  network: ${CLUSTER_NAME}
  deployment: {}
EOF

    Note that the gateway must still be created with a stable internal IP address, which is required for xDS communication with the istiod control plane in each cluster. NodePort peering is used for data plane communication, in that requests to services resolve to the NodePort instead of the gateway’s stable IP address.

    1. Create the istio-eastwest namespace.
        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
    2. Create the east-west gateway. For customization options, see the gateway guide in the Istio docs.
      • Solo distribution of istioctl: Create the east-west gateway. For more information about this command, see the CLI reference. Then, annotate the gateway so that cross-cluster traffic through this gateway resolves to the NodePort address.
          istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT} --generate > ew-gateway.yaml
kubectl apply -f ew-gateway.yaml --context ${CLUSTER_CONTEXT}
kubectl annotate gateway istio-eastwest -n istio-eastwest peering.solo.io/data-plane-service-type=NodePort --context ${CLUSTER_CONTEXT}
      • Helm: For more information about the peering chart, see the Helm values reference. For recommendations on customizing the east-west gateway for resiliency and availability with the Helm chart, see the best practices for multicluster peering.
          helm upgrade -i peering-eastwest oci://${HELM_REPO}/peering \
  --version ${ISTIO_IMAGE} \
  --namespace istio-eastwest \
  --kube-context ${CLUSTER_CONTEXT} \
  -f - <<EOF
eastwest:
  create: true
  cluster: ${CLUSTER_NAME}
  # The network that the istio-system namespace is labeled with.
  # In prod environments, network and cluster are likely not the same value.
  network: ${CLUSTER_NAME}
  dataplaneServiceTypes: nodeport
  deployment: {}
EOF

  5. Verify that the east-west gateway is successfully deployed.

      kubectl get pods -n istio-eastwest --context $CLUSTER_CONTEXT

  6. For each cluster that you want to include in the multicluster ambient mesh setup, repeat these steps to upgrade the ambient mesh components and create an east-west gateway in each cluster. Remember to change the cluster name and context variables each time you repeat the steps.

      export CLUSTER_NAME=<cluster-name>
export CLUSTER_CONTEXT=<cluster-context>

Deploy ambient components

If you do not have ambient meshes yet, use the Gloo Operator to deploy ambient mesh components to each cluster. Then, create east-west gateways so that traffic requests can be routed cross-cluster, and link clusters to enable cross-cluster service discovery.

  1. Save the name and kubeconfig context of a cluster where you want to install Istio in the following environment variables. Each time you repeat the steps in this guide, you change these variables to the next workload cluster’s name and context.

      export CLUSTER_NAME=<cluster-name>
export CLUSTER_CONTEXT=<cluster-context>

  2. Install the Gloo Operator to the gloo-mesh namespace. This operator deploys and manages your Istio installation. For more information, see the Helm reference. Note that if you already installed Solo Enterprise for Istio, you can optionally reference the secret that Solo Enterprise for Istio automatically creates for your license in the –set manager.env.SOLO_ISTIO_LICENSE_KEY_SECRET_REF=gloo-mesh/license-keys flag instead.

      helm install gloo-operator oci://us-docker.pkg.dev/solo-public/gloo-operator-helm/gloo-operator \
  --version 0.4.4 \
  -n gloo-mesh \
  --create-namespace \
  --kube-context ${CLUSTER_CONTEXT} \
  --set manager.env.SOLO_ISTIO_LICENSE_KEY=${SOLO_ISTIO_LICENSE_KEY}

  3. Verify that the operator pod is running.

      kubectl get pods -n gloo-mesh --context ${CLUSTER_CONTEXT} -l app.kubernetes.io/name=gloo-operator

    Example output:

      gloo-operator-78d58d5c7b-lzbr5     1/1     Running   0          48s
  4. Create a ServiceMeshController custom resource to configure an Istio installation. For a description of each configurable field, see the ServiceMeshController reference. If you need to set more advanced Istio configuration, you can also create a gloo-extensions-config configmap.
      kubectl apply -n gloo-mesh --context ${CLUSTER_CONTEXT} -f -<<EOF
apiVersion: operator.gloo.solo.io/v1
kind: ServiceMeshController
metadata:
  name: managed-istio
  labels:
    app.kubernetes.io/name: managed-istio
spec:
  cluster: ${CLUSTER_NAME}
  network: ${CLUSTER_NAME}
  dataplaneMode: Ambient
  installNamespace: istio-system
  version: ${ISTIO_VERSION}
EOF

  5. Verify that the ServiceMeshController is ready. In the Status section of the output, make sure that all statuses are True, and that the phase is SUCCEEDED.

      kubectl describe servicemeshcontroller -n gloo-mesh managed-istio --context ${CLUSTER_CONTEXT}

    Example output:

      ...
Status:
  Conditions:
    Last Transition Time:  2024-12-27T20:47:01Z
    Message:               Manifests initialized
    Observed Generation:   1
    Reason:                ManifestsInitialized
    Status:                True
    Type:                  Initialized
    Last Transition Time:  2024-12-27T20:47:02Z
    Message:               CRDs installed
    Observed Generation:   1
    Reason:                CRDInstalled
    Status:                True
    Type:                  CRDInstalled
    Last Transition Time:  2024-12-27T20:47:02Z
    Message:               Deployment succeeded
    Observed Generation:   1
    Reason:                DeploymentSucceeded
    Status:                True
    Type:                  ControlPlaneDeployed
    Last Transition Time:  2024-12-27T20:47:02Z
    Message:               Deployment succeeded
    Observed Generation:   1
    Reason:                DeploymentSucceeded
    Status:                True
    Type:                  CNIDeployed
    Last Transition Time:  2024-12-27T20:47:02Z
    Message:               Deployment succeeded
    Observed Generation:   1
    Reason:                DeploymentSucceeded
    Status:                True
    Type:                  WebhookDeployed
    Last Transition Time:  2024-12-27T20:47:02Z
    Message:               All conditions are met
    Observed Generation:   1
    Reason:                SystemReady
    Status:                True
    Type:                  Ready
  Phase:                   SUCCEEDED
Events:                    <none>

  6. Verify that the istiod control plane, Istio CNI, and ztunnel pods are running.

      kubectl get pods -n istio-system --context ${CLUSTER_CONTEXT}

    Example output:

      NAME                          READY   STATUS    RESTARTS   AGE
istio-cni-node-6s5nk          1/1     Running   0          2m53s
istio-cni-node-blpz4          1/1     Running   0          2m53s
istiod-gloo-bb86b959f-msrg7   1/1     Running   0          2m45s
istiod-gloo-bb86b959f-w29cm   1/1     Running   0          3m
ztunnel-mx8nw                 1/1     Running   0          2m52s
ztunnel-w8r6c                 1/1     Running   0          2m52s

  7. Apply the CRDs for the Kubernetes Gateway API to your cluster, which are required to create components such as waypoint proxies for L7 traffic policies, gateways with the Gateway resource, and more.

      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml --context ${CLUSTER_CONTEXT}

  8. Create an east-west gateway in the istio-eastwest namespace. In each cluster, the east-west gateway is implemented as a ztunnel that facilitates traffic between services across clusters in your multicluster mesh. You can use either LoadBalancer or NodePort addresses for cross-cluster traffic.

    1. Create the istio-eastwest namespace.
        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
    2. Create the east-west gateway. Cross-cluster traffic though this gateway resolves to the LoadBalancer address. For customization options, see the gateway guide in the Istio docs.
      • Solo distribution of istioctl: For more information about this command, see the CLI reference.
          istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT} --generate > ew-gateway.yaml
kubectl apply -f ew-gateway.yaml --context ${CLUSTER_CONTEXT}
      • Helm: For more information about the peering chart, see the Helm values reference. For recommendations on customizing the east-west gateway for resiliency and availability with the Helm chart, see the best practices for multicluster peering.
          helm upgrade -i peering-eastwest oci://${HELM_REPO}/peering \
  --version ${ISTIO_IMAGE} \
  --namespace istio-eastwest \
  --kube-context ${CLUSTER_CONTEXT} \
  -f - <<EOF
eastwest:
  create: true
  cluster: ${CLUSTER_NAME}
  # The network that the istio-system namespace is labeled with.
  # In prod environments, network and cluster are likely not the same value.
  network: ${CLUSTER_NAME}
  deployment: {}
EOF

    Note that the gateway must still be created with a stable internal IP address, which is required for xDS communication with the istiod control plane in each cluster. NodePort peering is used for data plane communication, in that requests to services resolve to the NodePort instead of the gateway’s stable IP address.

    1. Create the istio-eastwest namespace.
        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
    2. Create the east-west gateway. For customization options, see the gateway guide in the Istio docs.
      • Solo distribution of istioctl: Create the east-west gateway. For more information about this command, see the CLI reference. Then, annotate the gateway so that cross-cluster traffic through this gateway resolves to the NodePort address.
          istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT} --generate > ew-gateway.yaml
kubectl apply -f ew-gateway.yaml --context ${CLUSTER_CONTEXT}
kubectl annotate gateway istio-eastwest -n istio-eastwest peering.solo.io/data-plane-service-type=NodePort --context ${CLUSTER_CONTEXT}
      • Helm: For more information about the peering chart, see the Helm values reference. For recommendations on customizing the east-west gateway for resiliency and availability with the Helm chart, see the best practices for multicluster peering.
          helm upgrade -i peering-eastwest oci://${HELM_REPO}/peering \
  --version ${ISTIO_IMAGE} \
  --namespace istio-eastwest \
  --kube-context ${CLUSTER_CONTEXT} \
  -f - <<EOF
eastwest:
  create: true
  cluster: ${CLUSTER_NAME}
  # The network that the istio-system namespace is labeled with.
  # In prod environments, network and cluster are likely not the same value.
  network: ${CLUSTER_NAME}
  dataplaneServiceTypes: nodeport
  deployment: {}
EOF

  9. Verify that the east-west gateway is successfully deployed.

      kubectl get pods -n istio-eastwest --context $CLUSTER_CONTEXT

  10. For each cluster that you want to include in the multicluster ambient mesh setup, repeat these steps to install the Gloo Operator, ambient mesh components, and east-west gateway in each cluster. Remember to change the cluster name and context variables each time you repeat the steps.

      export CLUSTER_NAME=<cluster-name>
export CLUSTER_CONTEXT=<cluster-context>

Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters.

  1. Optional: Before you link clusters, you can check the individual readiness of each cluster for linking by running the istioctl multicluster check --precheck command. For more information about this command, see the CLI reference. If any checks fail, run the command with --verbose, and see Validate your multicluster setup.

      istioctl multicluster check --precheck --contexts="<context1>,<context2>,<context3>"

    Before continuing to the next step, make sure that the following checks pass as expected:
    ✅ Relevant environment variables on istiod are supported.
    ✅ The license in use by istiod supports multicluster.
    ✅ All istiod, ztunnel, and east-west gateway pods are healthy.
    ✅ The east-west gateway is programmed.

  2. Link clusters to enable cross-cluster service discovery and allow traffic to be routed through east-west gateways across clusters. Note that you can either link the clusters bi-directionally or asymmetrically. In a standard bi-directional setup, services in any of the linked clusters can send requests to and receive requests from the services in any of the other linked clusters. In an asymmetrical setup, you allow one cluster to send requests to another cluster, but the other cluster cannot send requests back to the first cluster.

    Use the istioctl multicluster link command to quickly link clusters.

    1. Verify that the contexts for the clusters that you want to include in the multicluster mesh are listed in your kubeconfig file, which is required for the istioctl multicluster link command. If you do not have access to the kubeconfig files, use the Helm or declarative resources tabs.

        kubectl config get-contexts
      • In the output, note the names of the cluster contexts, which you use in the next step to link the clusters.
      • If you have multiple kubeconfig files, you can generate a merged kubeconfig file by running the following command.
          KUBECONFIG=<kubeconfig_file1>.yaml:<file2>.yaml:<file3>.yaml kubectl config view --flatten

    2. Using the names of the cluster contexts, link the clusters so that they can communicate. To take a look at the Gateway resources that this command creates, you can include the --generate flag in the command. For more information about this command, see the CLI reference.

      • Bi-directional: You can use the following istioctl command to quickly link the clusters bi-directionally. In each cluster, Gateway resources are created that use the istio-remote GatewayClass. This class allows the gateways to connect to other clusters by using the addresses of the east-west gateways.

          istioctl multicluster link --namespace istio-eastwest --contexts="<context1>,<context2>,<context3>"

        If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, use the --preferred-data-plane-service-type nodeport flag when linking clusters. This automatically configures the peering gateways for NodePort-based cross-cluster traffic. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.

          istioctl multicluster link --namespace istio-eastwest --contexts="<context1>,<context2>,<context3>" --preferred-data-plane-service-type nodeport

        Example output for two clusters:

          Gateway istio-eastwest/istio-remote-peer-cluster1 applied to cluster "<cluster2_context>" pointing to cluster "<cluster1_context>" (network "cluster1")
Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "<cluster1_context>" pointing to cluster "<cluster2_context>" (network "cluster2")

      • Asymmetrical: You can use the following istioctl command to quickly link the clusters asymmetrically. The services in the cluster in the --from flag can send requests to services in the cluster in the --to flag, but sending requests in the reverse direction is not permitted.

          istioctl multicluster link --namespace istio-eastwest --from <context1> --to <context2>

        For example, this command allows services in cluster1’s mesh to send requests to services in cluster2’s mesh through cluster2’s east-west gateway. However, the reverse is not permitted: services in cluster2’s mesh cannot send requests through cluster1’s east-west gateway to services in cluster1.

          istioctl multicluster link --namespace istio-eastwest --from <cluster1_context> --to <cluster2_context>

        If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, use the --preferred-data-plane-service-type nodeport flag when linking clusters. This automatically configures the peering gateways for NodePort-based cross-cluster traffic. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.

          istioctl multicluster link --namespace istio-eastwest --from <cluster1_context> --to <cluster2_context> --preferred-data-plane-service-type nodeport

        Example output:

          Gateway istio-eastwest/istio-remote-peer-cluster2 applied to cluster "<cluster1_context>" pointing to cluster "<cluster2_context>" (network "cluster2")

    In the Solo distribution of Istio 1.29 or later, you can use the peering Helm chart to link your clusters. For more information about the peering chart, see the Helm values reference.

    1. Get the addresses of the east-west gateway in each cluster. The following commands show examples for two clusters.

        export CLUSTER1_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context $REMOTE_CONTEXT1 -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context $REMOTE_CONTEXT2 -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")

echo "Cluster1 east-west gateway: $CLUSTER1_EW_ADDRESS"
echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"

    2. Link the clusters by creating Helm releases in each cluster to represent the other clusters. The following example depicts a bi-directional setup. In an asymmetrical setup, you create Helm releases to only represent the directionality you want to allow. In each cluster where you create Helm releases, a Gateway resource is created that uses the istio-remote GatewayClass. This class allows the gateway to connect to other clusters by using the addresses of the east-west gateways.

      • Be sure to update the region, and optionally zone, of each cluster.
      • If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, change the preferredDataplaneServiceType to nodeport. The peering gateways must be created with a stable InternalIP node address, which is required for xDS communication with the istiod control plane in each cluster. When NodePort peering is enabled, it is used for data plane communication, in that requests to services resolve to the NodePort instead of the gateway’s stable IP address.
        helm upgrade -i peering-remote oci://${HELM_REPO}/peering \
  --version ${ISTIO_IMAGE} \
  --namespace istio-eastwest \
  --kube-context ${REMOTE_CONTEXT1} \
  -f - <<EOF
remote:
  create: true
  items:
    # Remote peer configuration for cluster2
    - name: istio-remote-peer-${REMOTE_CLUSTER2}
      cluster: ${REMOTE_CLUSTER2}
      network: ${REMOTE_CLUSTER2}
      # Change to "Hostname" if using hostname
      addressType: IPAddress
      address: ${CLUSTER2_EW_ADDRESS}
      # Change to "nodeport" if using NodePorts
      preferredDataplaneServiceType: loadbalancer
      trustDomain: cluster.local
      region: "<cluster2_region>"
      # Uncomment as needed
      # zone: "<cluster2_zone>"
EOF

helm upgrade -i peering-remote oci://${HELM_REPO}/peering \
  --version ${ISTIO_IMAGE} \
  --namespace istio-eastwest \
  --kube-context ${REMOTE_CONTEXT2} \
  -f - <<EOF
remote:
  create: true
  items:
    # Remote peer configuration for cluster1
    - name: istio-remote-peer-${REMOTE_CLUSTER1}
      cluster: ${REMOTE_CLUSTER1}
      network: ${REMOTE_CLUSTER1}
      # Change to "Hostname" if using hostname
      addressType: IPAddress
      address: ${CLUSTER1_EW_ADDRESS}
      # Change to "nodeport" if using NodePorts
      preferredDataplaneServiceType: loadbalancer
      trustDomain: cluster.local
      region: "<cluster1_region>"
      # Uncomment as needed
      # zone: "<cluster1_zone>"
 EOF

    Link the clusters by declaratively creating istio-remote peer gateways.

    Bi-directional: Use the following Gateway resources to create an istio-remote peer gateway in each cluster. The istio-remote GatewayClass allows the gateways to connect to other clusters by using the addresses of the east-west gateways.

    1. Get the addresses of the east-west gateway in each cluster. The following commands show examples for two clusters.

        export CLUSTER1_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context $REMOTE_CONTEXT1 -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context $REMOTE_CONTEXT2 -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")

echo "Cluster1 east-west gateway: $CLUSTER1_EW_ADDRESS"
echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"

    2. Using the east-west gateway addresses, create a Gateway resource in each cluster to represent the other cluster.

      • In the labels section, be sure to update the locality labels according to the region, and optionally zone, of each cluster. For more information about locality support, see the release notes.
      • If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, uncomment the peering.solo.io/preferred-data-plane-service-type: NodePort annotation from each Gateway resource. Additionally, you can comment out the HBONE listener in each gateway, because traffic is routed through the NodePort directly. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.
        kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gateway.istio.io/service-account: istio-eastwest
    gateway.istio.io/trust-domain: $REMOTE_CLUSTER2
    # Uncomment for NodePort-based cross-cluster traffic
    # peering.solo.io/preferred-data-plane-service-type: NodePort
  labels:
    topology.istio.io/network: $REMOTE_CLUSTER2
    topology.kubernetes.io/region: "<cluster2_region>"
    # Uncomment as needed
    # topology.kubernetes.io/zone: "<cluster2_zone>"
  name: istio-remote-peer-$REMOTE_CLUSTER2
  namespace: istio-eastwest
spec:
  addresses:
  # Change to 'type: Hostname' as needed, such as for AWS hostnames
  - type: IPAddress
    value: $CLUSTER2_EW_ADDRESS
  gatewayClassName: istio-remote
  listeners:
  # Comment HBONE out for NodePort-based cross-cluster traffic
  - name: cross-network
    port: 15008
    protocol: HBONE
    tls:
      mode: Passthrough
  - name: xds-tls
    port: 15012
    protocol: TLS
    tls:
      mode: Passthrough
EOF

kubectl apply --context $REMOTE_CONTEXT2 -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gateway.istio.io/service-account: istio-eastwest
    gateway.istio.io/trust-domain: $REMOTE_CLUSTER1
    # Uncomment for NodePort-based cross-cluster traffic
    # peering.solo.io/preferred-data-plane-service-type: NodePort
  labels:
    topology.istio.io/network: $REMOTE_CLUSTER1
    topology.kubernetes.io/region: "<cluster1_region>"
    # Uncomment as needed
    # topology.kubernetes.io/zone: "<cluster1_zone>"
  name: istio-remote-peer-$REMOTE_CLUSTER1
  namespace: istio-eastwest
spec:
  addresses:
  # Change to 'type: Hostname' as needed, such as for AWS hostnames
  - type: IPAddress
    value: $CLUSTER1_EW_ADDRESS
  gatewayClassName: istio-remote
  listeners:
  # Comment HBONE out for NodePort-based cross-cluster traffic
  - name: cross-network
    port: 15008
    protocol: HBONE
    tls:
      mode: Passthrough
  - name: xds-tls
    port: 15012
    protocol: TLS
    tls:
      mode: Passthrough
EOF

    Asymmetrical: Use the following Gateway resources to create an istio-remote peer gateway in only some clusters. The istio-remote GatewayClass allows the gateway in one cluster to connect to another cluster by using the address of the east-west gateway, but sending requests in the reverse direction is not permitted.

    1. Get the address of the east-west gateway in the cluster that you want to send traffic to. The following command shows an example for cluster2.

        export CLUSTER2_EW_ADDRESS=$(kubectl get svc -n istio-eastwest istio-eastwest --context $REMOTE_CONTEXT2 -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")

echo "Cluster2 east-west gateway: $CLUSTER2_EW_ADDRESS"

    2. Using the east-west gateway address, create a Gateway resource in the cluster that you want to send requests from. For example, this Gateway resource allows services in cluster1’s mesh to send requests to services in cluster2’s mesh through cluster2’s east-west gateway. However, the reverse is not permitted: services in cluster2’s mesh cannot send requests through cluster1’s east-west gateway to services in cluster1.

      • In the labels section, be sure to update the locality labels according to the region, and optionally zone, of each cluster. For more information about locality support, see the release notes.
      • If you want to use NodePorts instead of the gateway LoadBalancer IP address for cross-cluster traffic, uncomment the peering.solo.io/preferred-data-plane-service-type: NodePort annotation from each Gateway resource. Additionally, you can comment out the HBONE listener in each gateway, because traffic is routed through the NodePort directly. For more information about NodePort peering considerations and requirements, see Cross-cluster traffic addresses.
        kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gateway.istio.io/service-account: istio-eastwest
    gateway.istio.io/trust-domain: $REMOTE_CLUSTER2
    # Uncomment for NodePort-based cross-cluster traffic
    # peering.solo.io/preferred-data-plane-service-type: NodePort
  labels:
    topology.istio.io/network: $REMOTE_CLUSTER2
    topology.kubernetes.io/region: "<cluster2_region>"
    # Uncomment as needed
    # topology.kubernetes.io/zone: "<cluster2_zone>"
  name: istio-remote-peer-$REMOTE_CLUSTER2
  namespace: istio-eastwest
spec:
  addresses:
  # Change to 'type: Hostname' as needed, such as for AWS hostnames
  - type: IPAddress
    value: $CLUSTER2_EW_ADDRESS
  gatewayClassName: istio-remote
  listeners:
  # Comment HBONE out for NodePort-based cross-cluster traffic
  - name: cross-network
    port: 15008
    protocol: HBONE
    tls:
      mode: Passthrough
  - name: xds-tls
    port: 15012
    protocol: TLS
    tls:
      mode: Passthrough
EOF

  3. Verify that peer linking was successful by running the istioctl multicluster check command. If any checks fail, run the command with --verbose, and see Validate your multicluster setup.

      istioctl multicluster check --contexts="<context1>,<context2>,<context3>"

    In this example output, the remote peer gateways are successfully connected, the intermediate certificates are compatible between the clusters, each cluster has a unique, properly configured network, and no stale workloads were found because no autogenerated workload entries existed in the clusters prior to peering. If you do have preexisting autogenerated workload entries, the check verifies whether all entries are up to date.

      === Cluster: cluster1 ===
✅ Incompatible Environment Variable Check: all relevant environment variables are valid
✅ License Check: license is valid for multicluster
✅ Pod Check (istiod): all pods healthy
✅ Pod Check (ztunnel): all pods healthy
✅ Pod Check (eastwest gateway): all pods healthy
✅ Gateway Check: all eastwest gateways programmed
✅ Peers Check: all clusters connected
====== 

=== Cluster: cluster2 ===
✅ Incompatible Environment Variable Check: all relevant environment variables are valid
✅ License Check: license is valid for multicluster
✅ Pod Check (istiod): all pods healthy
✅ Pod Check (ztunnel): all pods healthy
✅ Pod Check (eastwest gateway): all pods healthy
✅ Gateway Check: all eastwest gateways programmed
✅ Peers Check: all clusters connected
====== 

✅ Intermediate Certs Compatibility Check: all clusters have compatible intermediate certificates
✅ Network Configuration Check: all network configurations are valid
⚠  Stale Workloads Check: no autogenflat workload entries found

  4. Optional: Verify that the istiod control plane for each peered cluster is included in each cluster’s proxy status list.

      istioctl proxy-status --context $REMOTE_CONTEXT1
istioctl proxy-status --context $REMOTE_CONTEXT2

    Example output for cluster-1, in which you can verify that the istiod control plane for cluster-2 is listed:

      NAME                                               CLUSTER          ISTIOD                      VERSION              SUBSCRIBED TYPES
istio-eastwest-67fd5679dc-fhsxs.istio-eastwest     cluster-1        istiod-7b7c9cc4c6-bdm9c     1.28.1-solo-fips     2 (WADS,WDS)
istiod-6bc6765484-5bbhd.istio-system               cluster-2        istiod-7b7c9cc4c6-bdm9c     1.28.1-solo-fips     3 (FSDS,SGDS,WDS)
ztunnel-5f8rb.kube-system                          cluster-1        istiod-7b7c9cc4c6-bdm9c     1.28.1-solo-fips     2 (WADS,WDS)
ztunnel-f96kh.kube-system                          cluster-1        istiod-7b7c9cc4c6-bdm9c     1.28.1-solo-fips     2 (WADS,WDS)
ztunnel-vtj4f.kube-system                          cluster-1        istiod-7b7c9cc4c6-bdm9c     1.28.1-solo-fips     2 (WADS,WDS)

Next: Add apps to the ambient mesh. For multicluster setups, this includes making specific services available across your linked cluster setup.

Option 2: Automatically link clusters (beta)

In each cluster, use the Gloo Operator to create the ambient mesh components, and create east-west gateways so that traffic requests can be routed cross-cluster. Then, use the Gloo management plane to automate multicluster linking, which enables cross-cluster service discovery.

Set up tools

  1. Set your Enterprise level license for Solo Enterprise for Istio as an environment variable. If you do not have one, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing. Note that you might have previously saved this key in another variable, such as ${SOLO_LICENSE_KEY} or ${GLOO_MESH_LICENSE_KEY}.

      export SOLO_ISTIO_LICENSE_KEY=<enterprise_license_key>

  2. Choose the version of Istio that you want to install or upgrade to by reviewing the supported versions.

  3. Save the Solo distribution of Istio version.

      export ISTIO_VERSION=1.28.1
export ISTIO_IMAGE=${ISTIO_VERSION}-solo

  4. Save the image and Helm repository information for the Solo distribution of Istio.

    • Istio 1.29 and later:

        export REPO=us-docker.pkg.dev/soloio-img/istio
export HELM_REPO=us-docker.pkg.dev/soloio-img/istio-helm

    • Istio 1.28 and earlier: You must provide a repo key for the minor version of the Solo distribution of Istio that you want to install. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Istio images built by Solo.io support article.

        # 12-character hash at the end of the repo URL
export REPO_KEY=<repo_key>
export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}

  5. Get the Solo distribution of Istio binary and install istioctl, which you use for multicluster linking and gateway commands. This script automatically detects your OS and architecture, downloads the appropriate Solo distribution of Istio binary, and verifies the installation.

      bash <(curl -sSfL https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/gloo-mesh/install-istioctl.sh)
export PATH=${HOME}/.istioctl/bin:${PATH}

Enable automatic peering of clusters

Upgrade Solo Enterprise for Istio in your multicluster setup to enable the ConfigDistribution feature flag and install the enterprise CRDs, which are required for Solo Enterprise for Istio to automate peering and distribute gateways between clusters.

  1. Upgrade your gloo-platform-crds Helm release in the management cluster to include the following settings.

      helm get values gloo-platform-crds -n gloo-mesh -o yaml --kube-context ${MGMT_CONTEXT} > mgmt-crds.yaml
helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \
    --kube-context ${MGMT_CONTEXT} \
    --namespace gloo-mesh \
    -f mgmt-crds.yaml \
    --set featureGates.ConfigDistribution=true \
    --set installEnterpriseCrds=true

  2. Upgrade your gloo-platform Helm release in the management cluster to include the following settings.

      helm get values gloo-platform -n gloo-mesh -o yaml --kube-context ${MGMT_CONTEXT} > mgmt-plane.yaml
helm upgrade gloo-platform gloo-platform/gloo-platform \
    --kube-context ${MGMT_CONTEXT} \
    --namespace gloo-mesh \
    -f mgmt-plane.yaml \
    --set featureGates.ConfigDistribution=true

  3. Upgrade your gloo-platform-crds Helm release in each workload cluster to include the following settings. Repeat this step for each workload cluster.

      helm get values gloo-platform-crds -n gloo-mesh -o yaml --kube-context ${CLUSTER_CONTEXT} > crds.yaml
helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \
    --kube-context ${CLUSTER_CONTEXT} \
    --namespace gloo-mesh \
    -f crds.yaml \
    --set installEnterpriseCrds=true

Create a shared root of trust

Create a shared root of trust for each cluster in the multicluster setup, including the management cluster. This can be achieved by providing a root certificate signed by a PKI provider, or a custom root certificate created for this purpose. The root certificate signs a unique intermediate CA certificate for each cluster.

By default, the Istio CA generates a self-signed root certificate and key, and uses them to sign the workload certificates. For more information, see the Plug in CA Certificates guide in the community Istio documentation.

For demo installations, you can run the following function to quickly generate and plug in the certificates and key for the Istio CA:

  curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}

mkdir -p certs
pushd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

function create_cacerts_secret() {
  context=${1:?context}
  cluster=${2:?cluster}
  make -f ../tools/certs/Makefile.selfsigned.mk ${cluster}-cacerts
  kubectl --context=${context} create ns istio-system || true
  kubectl --context=${context} create secret generic cacerts -n istio-system \
    --from-file=${cluster}/ca-cert.pem \
    --from-file=${cluster}/ca-key.pem \
    --from-file=${cluster}/root-cert.pem \
    --from-file=${cluster}/cert-chain.pem
}

create_cacerts_secret ${MGMT_CONTEXT} ${MGMT_CLUSTER}
create_cacerts_secret ${REMOTE_CONTEXT1} ${REMOTE_CLUSTER1}
create_cacerts_secret ${REMOTE_CONTEXT2} ${REMOTE_CLUSTER2}

cd ../..

To enhance the security of your setup even further and have full control over the Istio CA lifecycle, you can generate and store the root and intermediate CA certificates and keys with your own PKI provider. You can then use tools such as cert-manager to send certificate signing requests on behalf of istiod to your PKI provider. Cert-manager stores the signed intermediate certificates and keys in the cacerts Kubernetes secret so that istiod can use these credentials to issue leaf certificates for the workloads in the service mesh. You can set up cert-manager to also check the certificates and renew them before they expire.

AWS Private CA issuer and cert-manager: For an architectural overview of this certificate setup, see Bring your own Istio CAs with AWS. For steps on how to deploy this certificate setup, check out this Solo.io blog post. Be sure to repeat the steps so that a cacerts secret exists in each cluster.

Deploy ambient components

  1. Save the name and kubeconfig context of a cluster where you want to install Istio in the following environment variables. Each time you repeat the steps in this guide, you change these variables to the next workload cluster’s name and context. Note that to use automated multicluster peering, you must complete these steps to install an ambient mesh in the management cluster as well as your workload clusters.

      export CLUSTER_NAME=<cluster-name>
export CLUSTER_CONTEXT=<cluster-context>

  2. Apply the CRDs for the Kubernetes Gateway API to your cluster, which are required to create components such as waypoint proxies for L7 traffic policies, gateways with the Gateway resource, and more.

      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml --context ${CLUSTER_CONTEXT}

  3. Install the Gloo Operator to the gloo-mesh namespace. This operator deploys and manages your Istio installation. For more information, see the Helm reference. Note that if you already installed Solo Enterprise for Istio, you can optionally reference the secret that Solo Enterprise for Istio automatically creates for your license in the –set manager.env.SOLO_ISTIO_LICENSE_KEY_SECRET_REF=gloo-mesh/license-keys flag instead.

      helm install gloo-operator oci://us-docker.pkg.dev/solo-public/gloo-operator-helm/gloo-operator \
  --version 0.4.4 \
  -n gloo-mesh \
  --create-namespace \
  --kube-context ${CLUSTER_CONTEXT} \
  --set manager.env.SOLO_ISTIO_LICENSE_KEY=${SOLO_ISTIO_LICENSE_KEY}

  4. Verify that the operator pod is running.

      kubectl get pods -n gloo-mesh --context ${CLUSTER_CONTEXT} -l app.kubernetes.io/name=gloo-operator

    Example output:

      gloo-operator-78d58d5c7b-lzbr5     1/1     Running   0          48s

  5. Apply the following configmap and ServiceMeshController for the Gloo Operator to enable multicluster peering and deploy an ambient mesh.

      kubectl apply -n gloo-mesh --context ${CLUSTER_CONTEXT} -f -<<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: gloo-extensions-config
  namespace: gloo-mesh
data:
  beta: |
    serviceMeshController:
      multiClusterMode: Peering
  values.istiod: |
    env:
      PEERING_AUTOMATIC_LOCAL_GATEWAY: "true"	
EOF
---
kubectl apply -n gloo-mesh --context ${CLUSTER_CONTEXT} -f -<<EOF
apiVersion: operator.gloo.solo.io/v1
kind: ServiceMeshController
metadata:
  name: managed-istio
  labels:
    app.kubernetes.io/name: managed-istio
spec:
  cluster: ${CLUSTER_NAME}
  network: ${CLUSTER_NAME}
  dataplaneMode: Ambient
  installNamespace: istio-system
  version: ${ISTIO_VERSION}
EOF

  6. Verify that the istiod control plane, Istio CNI, and ztunnel pods are running.

      kubectl get pods -n istio-system --context ${CLUSTER_CONTEXT}

    Example output:

      NAME                          READY   STATUS    RESTARTS   AGE
istio-cni-node-6s5nk          1/1     Running   0          2m53s
istio-cni-node-blpz4          1/1     Running   0          2m53s
istiod-gloo-bb86b959f-msrg7   1/1     Running   0          2m45s
istiod-gloo-bb86b959f-w29cm   1/1     Running   0          3m
ztunnel-mx8nw                 1/1     Running   0          2m52s
ztunnel-w8r6c                 1/1     Running   0          2m52s

  7. Create an east-west gateway in the istio-eastwest namespace. In each cluster, the east-west gateway is implemented as a ztunnel that facilitates traffic between services across clusters in your multicluster mesh. You can use either LoadBalancer or NodePort addresses for cross-cluster traffic.

    1. Create the istio-eastwest namespace.
        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
    2. Create the east-west gateway. Cross-cluster traffic though this gateway resolves to the LoadBalancer address. For customization options, see the gateway guide in the Istio docs.
      • Solo distribution of istioctl: For more information about this command, see the CLI reference.
          istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT} --generate > ew-gateway.yaml
kubectl apply -f ew-gateway.yaml --context ${CLUSTER_CONTEXT}
      • Helm: For more information about the peering chart, see the Helm values reference. For recommendations on customizing the east-west gateway for resiliency and availability with the Helm chart, see the best practices for multicluster peering.
          helm upgrade -i peering-eastwest oci://${HELM_REPO}/peering \
  --version ${ISTIO_IMAGE} \
  --namespace istio-eastwest \
  --kube-context ${CLUSTER_CONTEXT} \
  -f - <<EOF
eastwest:
  create: true
  cluster: ${CLUSTER_NAME}
  # The network that the istio-system namespace is labeled with.
  # In prod environments, network and cluster are likely not the same value.
  network: ${CLUSTER_NAME}
  deployment: {}
EOF

    Note that the gateway must still be created with a stable internal IP address, which is required for xDS communication with the istiod control plane in each cluster. NodePort peering is used for data plane communication, in that requests to services resolve to the NodePort instead of the gateway’s stable IP address.

    1. Create the istio-eastwest namespace.
        kubectl create namespace istio-eastwest --context ${CLUSTER_CONTEXT}
    2. Create the east-west gateway. For customization options, see the gateway guide in the Istio docs.
      • Solo distribution of istioctl: Create the east-west gateway. For more information about this command, see the CLI reference. Then, annotate the gateway so that cross-cluster traffic through this gateway resolves to the NodePort address.
          istioctl multicluster expose --namespace istio-eastwest --context ${CLUSTER_CONTEXT} --generate > ew-gateway.yaml
kubectl apply -f ew-gateway.yaml --context ${CLUSTER_CONTEXT}
kubectl annotate gateway istio-eastwest -n istio-eastwest peering.solo.io/data-plane-service-type=NodePort --context ${CLUSTER_CONTEXT}
      • Helm: For more information about the peering chart, see the Helm values reference. For recommendations on customizing the east-west gateway for resiliency and availability with the Helm chart, see the best practices for multicluster peering.
          helm upgrade -i peering-eastwest oci://${HELM_REPO}/peering \
  --version ${ISTIO_IMAGE} \
  --namespace istio-eastwest \
  --kube-context ${CLUSTER_CONTEXT} \
  -f - <<EOF
eastwest:
  create: true
  cluster: ${CLUSTER_NAME}
  # The network that the istio-system namespace is labeled with.
  # In prod environments, network and cluster are likely not the same value.
  network: ${CLUSTER_NAME}
  dataplaneServiceTypes: nodeport
  deployment: {}
EOF

  8. Verify that the east-west gateway is successfully deployed.

      kubectl get pods -n istio-eastwest --context $CLUSTER_CONTEXT

  9. For each cluster that you want to include in the multicluster ambient mesh setup, including the management cluster, repeat these steps to install the Gloo Operator, ambient mesh components, and east-west gateway. Remember to change the cluster name and context variables each time you repeat the steps.

      export CLUSTER_NAME=<cluster-name>
export CLUSTER_CONTEXT=<cluster-context>

Review remote peer gateways

After you complete the steps for each cluster, verify that Solo Enterprise for Istio successfully created and distributed the remote peering gateways. These gateways use the istio-remote GatewayClass, which allows the istiod control plane in each cluster to discover the east-west gateway addresses of other clusters. Solo Enterprise for Istio generates one istio-remote resource in the management cluster for each connected workload cluster, and then distributes the gateway to each cluster respectively.

  1. Verify that an istio-remote gateway for each connected cluster is copied to the management cluster.

      kubectl get gateways -n istio-eastwest --context $MGMT_CONTEXT

    In this example output, the istio-remote gateways that were auto-generated for workload clusters cluster1 and cluster2 are copied to the management cluster, alongside the management cluster’s own istio-remote gateway and east-west gateway.

      NAMESPACE        NAME                            CLASS           ADDRESS                                                                   PROGRAMMED   AGE
istio-eastwest   istio-eastwest                 istio-eastwest   a7f6f1a2611fc4eb3864f8d688622fd4-1234567890.us-east-1.elb.amazonaws.com   True         6s
istio-eastwest   istio-remote-peer-cluster1     istio-remote     a5082fe9522834b8192a6513eb8c6b01-0987654321.us-east-1.elb.amazonaws.com   True         4s
istio-eastwest   istio-remote-peer-cluster2     istio-remote     aaad62dc3ffb142a1bfc13df7fe9665b-5678901234.us-east-1.elb.amazonaws.com   True         4s
istio-eastwest   istio-remote-peer-mgmt         istio-remote     a7f6f1a2611fc4eb3864f8d688622fd4-1234567890.us-east-1.elb.amazonaws.com   True         4s

  2. In each cluster, verify that all istio-remote gateways are successfully distributed to all workload clusters. This ensures that services in each workload cluster can now access the east-west gateways in other clusters of the multicluster mesh setup.

      kubectl get gateways -n istio-eastwest --context $CLUSTER_CONTEXT

  3. In each cluster, verify that peer linking was successful by running the istioctl multicluster check command. For more information about this command, see the CLI reference. If any checks fail, run the command with --verbose, and see Validate your multicluster setup.

      istioctl multicluster check --context $CLUSTER_CONTEXT

    Example output for one cluster:

      ✅ Incompatible Environment Variable Check: all relevant environment variables are valid
✅ License Check: license is valid for multicluster
✅ Pod Check (istiod): all pods healthy
✅ Pod Check (ztunnel): all pods healthy
✅ Pod Check (eastwest gateway): all pods healthy
✅ Gateway Check: all eastwest gateways programmed
✅ Peers Check: all clusters connected
====== 

ℹ️ Intermediate Certs Compatibility Check: root certificate SHA256 sum: 909190c7c1eb242cfaf53aee23f484dba85105d5a259b141017e92fb9a0ce1b3
✅ Network Configuration Check: all network configurations are valid
⚠️ Stale Workloads Check: skipped (requires contexts for multiple clusters)

Next

  • Add apps to the ambient mesh. For multicluster setups, this includes making specific services available across your linked cluster setup.
  • In a multicluster mesh, the east-west gateway serves as a ztunnel that allows traffic requests to flow across clusters, but it does not modify requests in any way. To control in-mesh traffic, you can instead apply policies to waypoint proxies that you create for a workload namespace.

Optional: Validate your multicluster setup

Both before and after you link clusters into a multicluster mesh, you can use the istioctl multicluster check command, along with other observability checks, to verify multiple aspects of multicluster ambient mesh support and status.

istioctl multicluster check

You can use the istioctl multicluster check --precheck command to check the individual readiness of each cluster before running istioctl multicluster link to link them in a multicluster mesh, and run it again after linking to confirm that the connections were successful. This command performs checks listed in the following sections, which you can review to understand what each check validates. Additionally, if any of the checks fail, run the command with the --verbose option, and review the following troubleshooting recommendations.

  istioctl multicluster check --verbose --contexts="<context1>,<context2>,<context3>"

For more information about this command, see the CLI reference.

Incompatible environment variables

Checks whether the ENABLE_PEERING_DISCOVERY=true and optionally K8S_SELECT_WORKLOAD_ENTRIES=true environment variables are set incorrectly or are not supported for multicluster ambient mesh.

Example verbose output:

  --- Incompatible Environment Variable Check ---

✅ Incompatible Environment Variable Check: K8S_SELECT_WORKLOAD_ENTRIES is valid ("")
✅ Incompatible Environment Variable Check: ENABLE_PEERING_DISCOVERY is valid ("true")
✅ Incompatible Environment Variable Check: all relevant environment variables are valid

If this check fails, check your environment variables in your istiod configuration, such as by running helm get values --kube-context ${CLUSTER_CONTEXT} istiod -n istio-system -o yaml, and update your configuration.

License validity

Checks whether the license in use by istiod is valid for multicluster ambient mesh. Multicluster capabilities require an Enterprise level license for Solo Enterprise for Istio.

Example verbose output:

  --- License Check ---

✅ License Check: license is valid for multicluster

If your license does not support multicluster ambient mesh, contact your Solo account representative.

Pod health

Checks the health of the pods in the cluster. All istiod, ztunnel, and east-west gateway pods across the checked clusters must be healthy and running for the multicluster mesh to function correctly.

Example verbose output:

  --- Pod Check (istiod) ---

NAME                        READY     STATUS      RESTARTS     AGE
istiod-6d9cdf88cf-l47tf     1/1       Running     0            10m18s

✅ Pod Check (istiod): all pods healthy


--- Pod Check (ztunnel) ---

NAME              READY     STATUS      RESTARTS     AGE
ztunnel-dvlwk     1/1       Running     0            10m6s

✅ Pod Check (ztunnel): all pods healthy


--- Pod Check (eastwest gateway) ---

NAME                                READY     STATUS      RESTARTS     AGE
istio-eastwest-857b77fc5d-qgnrl     1/1       Running     0            9m33s

✅ Pod Check (eastwest gateway): all pods healthy

To check any unhealthy pods, run the following commands. Consider checking the pod logs, and review Debug Istio.

  kubectl get po -n istio-system
kubectl get po -n istio-eastwest

East-west gateway status

Checks the status of the east-west gateways in the cluster. When an east-west gateway is created, the gateway controller creates a Kubernetes service to expose the gateway. Once this service is correctly attached to the gateway and has an address assigned, the east-west gateway has a Programmed status of true.

Example verbose output:

  --- Gateway Check ---

Gateway: istio-eastwest
Addresses:
- 172.18.7.110
Status: programmed ✅

✅ Gateway Check: all eastwest gateways programmed

If the Programmed status is not true, an issue might exist with the address allocation for the service. Check the east-west gateway with a command such as kubectl get svc -n istio-eastwest, and verify that your cloud provider can correctly allocate addresses to the service.

Remote peer gateway status

Checks the status of the remote peer gateways in the cluster, which represent the other peered clusters in the multicluster setup. These remote gateways configure the connection between the local cluster’s istiod control plane, and the peered clusters’ remote networks to enable xDS communication between peers. When the initial network connection between istiod and a remote peer is made, the gateway’s gloo.solo.io/PeerConnected status updates to true. Then, when the full xDS sync occurs between peers, the gateway’s gloo.solo.io/PeeringSucceeded status also updates to true.

Example verbose output:

  --- Peers Check ---

Cluster: cluster2
Addresses:
- 172.18.7.130
Conditions:
- Accepted: True
- Programmed: True
- gloo.solo.io/PeerConnected: True
- gloo.solo.io/PeeringSucceeded: True
- gloo.solo.io/PeerDataPlaneProgrammed: True
Status: connected ✅

✅ Peers Check: all clusters connected

If the connection is severed between the peers, the gloo.solo.io/PeerConnected status becomes false. A failed connection between peers can be due to either a misconfiguration in the peering setup, or a network issue blocking port 15008 on the remote cluster, which is the cross-network HBONE port that the east-west gateway listens on. Review the steps you took to link clusters together, such as the steps outlined in the Helm default network guide. Additionally, review any firewall rules or network policies that might block access through port 15008 on the remote cluster.

Intermediate certificate compatibility

Confirms the certificate compatibility between peered clusters. This check reads the root-cert.pem from the istio-ca-root-cert configmap in the istio-system namespace, and uses x509 certificate validation to confirm the root cert is compatible with all of the clusters’ ca-cert.pem intermediate certificate chains from the cacerts secret.

Example verbose output:

  --- Intermediate Certs Compatibility Check ---

ℹ  Intermediate Certs Compatibility Check: cluster cluster1 root certificate SHA256 sum: 6d18f32e134824c158d97f32618657c45d5a83839f838ada751757139481537e
ℹ  Intermediate Certs Compatibility Check: cluster cluster2 root certificate SHA256 sum: 6d18f32e134824c158d97f32618657c45d5a83839f838ada751757139481537e
✅ Intermediate Certs Compatibility Check: cluster cluster1 has compatible intermediate certificates with cluster cluster2 
✅ Intermediate Certs Compatibility Check: cluster cluster2 has compatible intermediate certificates with cluster cluster1 
✅ Intermediate Certs Compatibility Check: all clusters have compatible intermediate certificates

If this check fails because the root certs are not valid for each peered clusters’ intermediate certificate chain, you can check the istiod logs for TLS errors when attempting to communicate with a peered cluster, such as the following:

  2025-12-04T22:09:22.474517Z     warn    deltaadsc       disconnected, retrying in 24.735483751s: delta stream: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: remote error: tls: unknown certificate authority"       target=peering-cluster2

Ensure each cluster has a cacerts secret in the istio-system namespace. To regenerate invalid certificates for each cluster, follow the example steps in Create a shared root of trust.

Network configuration

Confirms the network configuration of the multicluster mesh. For multicluster peering setups that do not use a flat network topology, each cluster must occupy a unique network. The network name must be defined with the label topology.istio.io/network and set on both the istio-system namespace and the istio-eastwest gateway resource. The same network name must also be set as the NETWORK environment variable on the ztunnel daemonset. Each remote gateway that represents that cluster must have the topology.istio.io/network label equal to the network of the remote cluster.

Example verbose output:

  --- Network Configuration Check ---

✅ Cluster cluster1 has network: cluster1
✅ Eastwest gateway istio-eastwest/istio-eastwest has correct network label: cluster1
✅ Cluster cluster2 has network: cluster2
✅ Eastwest gateway istio-eastwest/istio-eastwest has correct network label: cluster2
✅ Remote gateway istio-eastwest/istio-remote-peer-cluster2 references network cluster2 (clusters: [cluster2])
✅ Remote gateway istio-eastwest/istio-remote-peer-cluster1 references network cluster1 (clusters: [cluster1])
✅ Network Configuration Check: all network configurations are valid

Mismatched network identities cause errors in cross-cluster communication, which leads to error logs in ztunnel pods that indicate a network timeout on the outbound communication. Notably, the destination address on these errors is a 240.X.X.X address, instead of the correct remote peer gateway address. You can run kubectl logs -l app=ztunnel -n istio-system --tail=10 --context ${CLUSTER_CONTEXT} | grep -iE "error|warn" to review logs such as the following:

  2025-11-18T16:14:53.490573Z     error   access  connection complete     src.addr=240.0.2.27:46802 src.workload="ratings-v1-5dc79b6bcd-zm8v6" src.namespace="bookinfo" src.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings" dst.addr=240.0.9.43:15008 dst.hbone_addr=240.0.9.43:9080 dst.service="productpage.bookinfo.mesh.internal" dst.workload="autogenflat.portfolio1-soloiopoc-cluster1.bookinfo.productpage-v1-54bb874995-hblwp.ee508601917c" dst.namespace="bookinfo" dst.identity="spiffe://cluster.local/ns/bookinfo/sa/bookinfo-productpage" direction="outbound" bytes_sent=0 bytes_recv=0 duration="10001ms" error="connection timed out, maybe a NetworkPolicy is blocking HBONE port 15008: deadline has elapsed"

To troubleshoot these issues, be sure that you use unique network names to represent each cluster, and that you correctly labeled the cluster’s istio-system namespace with that network name, such as by running kubectl label namespace istio-system --context ${CLUSTER_CONTEXT} topology.istio.io/network=${CLUSTER_NAME}. You can also relabel the east-west gateway in the cluster, and the remote peer gateways in other clusters that represent this cluster.

Stale workload entries

In flat network setups, checks for any outdated workload entries that must be removed from the multicluster mesh. Stale workload entries might exist from pods that were deleted, but the autogenerated entries for those workloads were not correctly cleaned up. If you do not use a flat network topology, no autogenerated workload entries exist to be validated, and this check can be ignored.

Example verbose output for a non-flat network setup:

  --- Stale Workloads Check ---

⚠  Stale Workloads Check: no autogenflat workload entries found

If you use a flat network topology, and this check fails with stale workload entries, run kubectl get workloadentries -n istio-system | grep autogenflat to list the autogenerated workload entries in the remote cluster, and compare the list to the output of kubectl get pods in the source cluster for those workloads. You can safely manually delete the stale workload entries in the remote cluster for pods that no longer exist in the source cluster, such as by running kubectl get workloadentries -n istio-system <entry_name>.

Metrics

You can also check metrics that are built into the Solo distribution of Istio to verify multiple aspects of the multicluster peering status.

Each peering metric has the labels source and peer, which appear as fields in the metric. The source is the local istiod instance in the cluster where the metric is emitted, and peer is the peered remote cluster. The convergence time metrics are important for understanding how quickly configuration propagates to peer clusters. A high convergence time can indicate slow propagation, connectivity issues, or that the peer or network is under load.

You can access the following metrics by using the Prometheus server that is built into the Solo Enterprise for Istio management plane. For setup steps, see the multicluster management plane guide. For more information, see the built-in Prometheus overview and sample PromQL queries.

If you use Grafana to monitor Istio performance, you can also check out the Grafana dashboards in the Solo Communities of Practice (COP) repository. For example, you can use the istio-peering-dashboard to monitor and verify peering connection between clusters.

MetricDescription
peer_connection_stateThe connection state of peered remote clusters (1 = connected, 0 = disconnected).
peer_convergence_time_bucketThe cumulative count of convergence times, which measures the delay between sending an xDS request to a peer cluster and receiving an ACK or NACK. This metric is captured in seconds for the following intervals (buckets): 0.01, 0.1, 0.5, 1, 3, 5, 10, 20, 30.
peer_convergence_time_countThe total number of xDS requests to peer clusters for which an ACK or NACK was received since istiod was last started.
peer_convergence_time_sumThe sum of all convergence times in seconds since istiod was last started.
peer_xds_config_size_bytes_bucketThe distribution of xDS configuration sizes received from peer clusters.
peer_xds_config_size_bytes_countThe number of xDS configurations received from peer clusters.
peer_xds_config_size_bytes_sumThe sum of all xDS configuration sizes received from peer clusters since the last start of the Istio proxy.

Further debugging and observability

For additional guidance around observing your multicluster ambient mesh, check out the observability overview, which contains links to guides on using logs, metrics, and traces in your Istio environment.

For additional guidance around debugging your multicluster ambient mesh, check out the Istio troubleshooting guide.

ServiceMeshController reference

Review the commonly configured fields for the ServiceMeshController custom resource. For the full list of available options, see the ServiceMeshController API reference.

SettingDescriptionSupported valuesDefault
clusterThe name of the cluster to install Istio into. This value is required to set the trust domain field in multicluster environments.
dataplaneModeThe dataplane mode to use.Ambient or SidecarAmbient
distributionOptional: A specific distribution of the Istio version, such as the standard or FIPS image distribution.Standard or FIPSStandard
image.repositoryOptional: An Istio image repository, such as to use an image from a private registry.The Solo distribution of Istio repo for the Istio minor version.
image.secretsOptional: A list of secrets to use for pulling images from a container registry. The secret list must be of type kubernetes.io/dockerconfigjson and exist in the installNamespace that you install Istio in.
installNamespaceNamespace to install the service mesh components into. If you set the installNamespace to a namespace other than gloo-system, gloo-mesh, or istio-system, you must include the –set manager.env.WATCH_NAMESPACES=<namespace> setting.istio-system
networkThe default network where workload endpoints exist. A network is a logical grouping of workloads that exist in the same Layer 3 domain. Workloads in the same network can directly communicate with each other, while workloads in different networks require an east-west gateway to establish connectivity. This value is required in multi-network environments. For example, an easy way to identify the network of in-mesh workloads in one cluster is to simply use the cluster’s name for the network, such as cluster1.
onConflictOptional: How to resolve conflicting Istio configuration, if the configuration in this ServiceMeshController conflicts with existing Istio resources in the cluster.
  • Force: The existing resources are updated with the new configuration.
  • Abort: The installation configured in this ServiceMeshController is aborted, and the existing resources remain unchanged.
Force or AbortAbort
repository.secretsOptional: A list of secrets to use for pulling manifests from an artifact registry. The secret list must be of type kubernetes.io/dockerconfigjson and can exist in any namespace, such as the same namespace that you create the ServiceMeshController in.
repository.insecureSkipVerifyOptional: If set to true, the repository server’s certificate chain and hostname are not verified.true or false
scalingProfileOptional: The istiod control plane scaling settings to use. In large environments, set to Large.
  • Default sets the following scaling values:
    • resources.requests.cpu=1000m
    • resources.requests.memory=1Gi
    • resources.limits.cpu=2000m
    • resources.limits.memory=2Gi
    • autoscaleEnabled=true
    • autoscaleMin=2
    • autoscaleMax=25
    • cpu.targetAverageUtilization=80
  • Demo sets the following scaling values:
    • autoscaleEnabled=false
    • resources.requests.cpu=250m
  • Large sets the following scaling values:
    • resources.requests.cpu=4000m
    • resources.requests.memory=4Gi
    • resources.limits.cpu=4000m
    • resources.limits.memory=4Gi
    • autoscaleEnabled=true
    • autoscaleMin=4
    • autoscaleMax=50
    • cpu.targetAverageUtilization=75
Default, Demo, or LargeDefault
trafficCaptureModeOptional: Traffic capture mode to use.
  • Auto: The most suitable traffic capture mode is automatically selected based on the environment, such as using a CNI to capture traffic.
  • InitContainer: An init container is used for the traffic capture. This setting can only be used when the dataplaneMode is Sidecar.
Auto or InitContainerAuto
trustDomainThe trustDomain for Istio workloads.If cluster is set, defaults to that value. If cluster is unset, defaults to cluster.local.
versionThe Istio patch version to install. For more information, see Supported Solo distributions of Istio.Any Istio version supported for your Gloo version

Advanced settings configuration

You can set advanced Istio configuation by creating a configmap. For example, you might need to specify settings for istiod such as discovery selectors, pod and service annotations, affinities, tolerations, or node selectors.

The following gloo-extensions-config example configmap sets all possible fields for demonstration purposes. Note that in some guides in this documentation set, Helm extension settings such as data.values.istiod are defined for specific settings in the configmap. However, these settings are used only when necessary, and are not recommended for other general use cases.

  apiVersion: v1
kind: ConfigMap
metadata:
  name: gloo-extensions-config
  namespace: gloo-mesh
data:
  stable: |
    serviceMeshController:
      istiod:
        discoverySelectors:
          - matchLabels:
              foo: bar
        topology:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                    - key: foo
                      operator: In
                      values:
                      - bar
                  topologyKey: foo.io/bar
                weight: 80
          nodeSelector:
            foo: bar
          tolerations:
            - key: t1
              operator: Equal
              value: v1
        deployment:
          podAnnotations:
            foo: bar
        serviceAnnotations:
          foo: bar
  beta: |
    serviceMeshController:
      cni:
        confDir: /foo/bar
        binDir: /foo/bar