Navigation :

Route across clusters

You can easily set up intelligent multi-cluster routing for active-active and active-passive workloads in your service mesh by using virtual destinations. With virtual destinations, you can define unique internal hostnames for apps that are spread across multiple clusters, and enable service discovery for these apps within the mesh by adding the hostnames to the service mesh registry.

Multi-cluster routing with Gloo Mesh

For more information, see the following resources:

The instructions in this guide assume that the services you want to route requests to are included in your service mesh. If you want to route to services or endpoints that are not included in your service mesh, see Route to services external to the mesh.

Before you begin

Install Gloo Mesh, register workload clusters, and install Istio into each workload cluster. For example, you might follow the demo setup to get a testing environment up and running, or the docs in the installation guide to set up your production-level environment. To enable multicluster routing, ensure that you deploy an east-west gateway.
In the workspace settings for each workspace that your services are in, ensure that you configure the following:
- Enable federation so that services in different clusters can communicate with each other.
- If you enable service isolation, services cannot communicate with services outside the mesh or in another workspace by default. If you want to set up multi-cluster routing across multiple workspaces, be sure to export the required resources from one workspace to the other workspaces that must access them. For example, if your ingress gateway is in a different workspace than your apps, be sure to export the app workspace resources to the gateway workspace.

Create a Gloo Mesh root trust policy to ensure that services in one cluster securely communicate with the services in other clusters. The root trust policy sets up the domain and certificates to establish a shared trust model across multiple clusters in your service mesh.

kubectl apply --context=$MGMT_CONTEXT -f - <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: root-trust
  namespace: gloo-mesh
spec:
  config:
    autoRestartPods: true
    mgmtServerCa:
      generated: {}
EOF

Follow the other guides in this routing section to plan your routing table setup. For example, you might check out the path matching guide to decide how to match the incoming requests to your service paths, the redirect guide to set up any path or host rewrites, or the sub-table delegation guide to nest and sort multiple route tables.

Set up multi-cluster routing for north-south traffic

To route incoming requests from your ingress gateway to an app that is spread across clusters based on locality, you create a virtual destination for your multicluster app, and a route table that forwards traffic to that virtual destination.

In the first cluster where you deployed an instance of your app, create a virtual gateway. This virtual gateway selects the default Istio ingress gateway, which routes incoming traffic (north-south) to your service mesh.

kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualGateway
metadata:
  name: ingress-gateway
  namespace: gloo-mesh
spec:
  workloads:
    # Selects the istio ingress gateway in workload cluster 1
    - selector:
        labels:
          istio: ingressgateway
        cluster: ${REMOTE_CLUSTER1}
  listeners:
    # The port the ingress gateway listens on for incoming requests to route
    - port:
        number: 80
      http: {}
EOF

Save the external address of the Istio ingress gateway.

GKE: LoadBalancer IP address
EKS: LoadBalancer hostname


   export CLUSTER_1_INGRESS_ADDRESS=$(kubectl --context $REMOTE_CONTEXT1 get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   echo $CLUSTER_1_INGRESS_ADDRESS


   export CLUSTER_1_INGRESS_ADDRESS=$(kubectl --context $REMOTE_CONTEXT1 get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   echo $CLUSTER_1_INGRESS_ADDRESS

In each cluster where you deployed an instance of your global app, ensure that the app is exposed by a Kubernetes service. Make sure that you use the same label for your services as this label serves as the selector to later identify all the services across your clusters that make up your global app. In this example, the label app: global-app is used. The service listens on port 3456 and forwards requests to port 9080.
```
apiVersion: v1
kind: Service
metadata:
  labels:
    app: global-app
  name: global-app
  namespace: global
spec:
  ports:
  - name: http
    port: 3456
    protocol: TCP
    targetPort: 9080
  type: ClusterIP
```
Create a virtual destination resource and define a unique hostname that the virtual gateway can use to send requests to. This virtual destination is configured to listen for incoming traffic on the internal-only, arbitrary hostname app.mesh.internal.com:8080. Incoming requests can then be routed to any service instances with the label app: global-app on port 9080. Note that because virtual destinations are dynamic, the ingress gateway that handles the request routes it to the closest healthy app instance.
```
kubectl apply --context $REMOTE_CONTEXT1 -n global -f- <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualDestination
metadata:
  name: global-app-vd
  namespace: global
spec:
  hosts:
  # Arbitrary, internal-only hostname assigned to the endpoint
  - app.mesh.internal.com
  ports:
  - number: 8080
    protocol: HTTP
    targetPort:
      number: 9080
  services:
    - labels:
        app: global-app
EOF
```

Create a route table to route requests to your app's services that are in any cluster in your service mesh. A route table allows you to define how requests to endpoints should be routed, and is translated to the Istio VirtualService resource. In this example route table, all requests to the /global-app path are routed to the global-app virtual destination.

kubectl apply --context $REMOTE_CONTEXT1 -n global -f- <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
  name: global-app-routes
  namespace: global
spec:
  # Applies to any host; can indicate a specific domain, like example.com
  hosts:
    - '*'
  # Selects the virtual gateway you previously created
  virtualGateways:
    - name: ingress-gateway
      namespace: gloo-mesh
      cluster: ${REMOTE_CLUSTER1}
  http:
    # Route for the global-app service
    - name: global-app
      # Prefix matching
      matchers:
      - uri:
          prefix: /global-app
      # Forwarding directive
      forwardTo:
        destinations:
          # Reference to the virtual destination for the global-app svcs
          - ref:
              name: global-app-vd
            kind: VIRTUAL_DESTINATION
            port:
              number: 8080
EOF

Test the route to your app by curling the ingress gateway address and app path. For example, the following command appends /global-app for the sample app.
```
curl http://$CLUSTER_1_INGRESS_ADDRESS/global-app
```

When you use a virtual destination, any request to the virtual destination is routed directly to the closest healthy app instance without leaving the service mesh. This is often the app instance in the same cluster, if available. If you need to specifically test access to the service in another cluster, you can apply a fault injection policy to purposefully fail the connection to the service in the first cluster.

Set up multi-cluster routing for east-west traffic

To route in-mesh requests from one app to another app that is spread across clusters, you create a virtual destination for your multicluster app. Then, you create a route table that forwards east-west traffic from the first app to that virtual destination.

Create a virtual destination resource for your second app that receives requests from the first app. This virtual destination is configured to listen for incoming traffic on the internal-only, arbitrary hostname second-app.mesh.internal.com:8080. Incoming requests can then be routed to any service instances with the label app: second-app on port 9080. Note that because virtual destinations are dynamic, the east-west gateway that handles the request routes it to the closest healthy app instance.

kubectl apply --context $REMOTE_CONTEXT1 -n global -f- <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualDestination
metadata:
  name: second-app-vd
  namespace: global
spec:
  hosts:
  # Arbitrary, internal-only hostname assigned to the endpoint
  - second-app.mesh.internal.com
  ports:
  - number: 8080
    protocol: HTTP
    targetPort:
      number: 9080
  services:
    - labels:
        app: second-app
EOF

Route requests initiated from your first app to your second app by using the virtual destination hostname. The routing method depends on whether your team can change the address that the first app sends requests to.

Change first app to route to new hostname (preferred)
Reroute existing traffic patterns

If you can change the initiator app's code, edit the app to call the second app using the virtual destination hostname instead of the service's DNS entry. In this setup, no route table is needed, because the initiator app can now directly call the virtual destination. However, if you want to also add routing rules for this virtual destination, you can create a route table, such as the following.

For hosts, specify the virtual destination hostname that you created in the previous step. In this example, second-app.mesh.internal.com is used.
For workloadSelectors, specify the first app that initiates the request to the second app. To instead select all workloads, do not specify this section. In this example, an app with the label app: global-app is selected.
For the http route, specify the virtual destination for the second app, and any routing rules you want to apply. For example, you might use weighted routing to the subset of app versions.

kubectl apply --context $REMOTE_CONTEXT1 -n global -f- <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
  name: second-app-routes
  namespace: global
spec:
  hosts:
  # Applies to the VD hostname that second-app listens on
    - 'second-app.mesh.internal.com'
  # Applies to requests sent by this initiator app
  workloadSelectors:
  - selector:
      labels:
        app: global-app
  http:
    # Route for the second-app service
    - name: second-app
      # Prefix matching
      matchers:
      - uri:
          prefix: /second-app
      # Forwarding directive
      forwardTo:
        destinations:
          # Reference to the virtual destination that directs 15% of reviews traffic to second-app-v1
          - ref:
              name: second-app-vd
            kind: VIRTUAL_DESTINATION
            port:
              number: 8080
            subset:
              version: v1
            weight: 15
          # Reference to the virtual destination that directs 85% of reviews traffic to second-app-v2
          - ref:
              name: second-app-vd
            kind: VIRTUAL_DESTINATION
            port:
              number: 8080
            subset:
              version: v2
            weight: 85
EOF

If you cannot change the initiator app to call the second app using the virtual destination hostname instead of the service's DNS entry, you can create a route table that defines how east-west requests within your mesh from the first app to the virtual destination for the second app should be routed.

For hosts, specify the internal DNS entry that the second app listens on, formatted such as <service-name>.<namespace-name>.svc.cluster.local. In this example, second-app.global.svc.cluster.local is used. The east-west gateway in your mesh does the work of taking requests made to the second-app.global.svc.cluster.local hostname and routing them to the second-app.mesh.internal.com virtual destination hostname that you specified in the previous step.
For workloadSelectors, specify the first app that initiates the request to the second app. To instead select all workloads, do not specify this section. In this example, an app with the label app: global-app is selected.
For the http route, specify the virtual destination for the second app.

kubectl apply --context $REMOTE_CONTEXT1 -n global -f- <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
  name: second-app-routes
  namespace: global
spec:
  hosts:
  # Applies to the internal hostname that second-app listens on
    - 'second-app.global.svc.cluster.local'
  # Applies to requests sent by this initiator app
  workloadSelectors:
  - selector:
      labels:
        app: global-app
  http:
    # Route for the second-app service
    - name: second-app
      # Forwarding directive
      forwardTo:
        destinations:
          # Reference to the virtual destination for the second-app svcs
          - ref:
              name: second-app-vd
            kind: VIRTUAL_DESTINATION
            port:
              number: 8080
EOF

Test the route from the first app to your second app. For example, log in to your first app and run nslookup second-app.mesh.internal.com to verify that the second app is reachable through the virtual destination hostname. Or, if you can access your first app externally, you can curl the ingress gateway address and the path for your first app, and verify that information from the second app is being successfully returned.

Example multi-cluster route table setup for BookInfo

To test out both north-south and east-west multi-cluster routing, you can follow the Bookinfo multi-cluster guide.

Complete the demo setup to install Gloo Mesh, Istio, and Bookinfo in your cluster.
Follow Step 3: Expose the ingress gateway in the Multicluster federation and isolation with Bookinfo guide. These steps walk you through creating a virtual gateway and configuring a north-south route table for the BookInfo services in cluster-1, which you can then test access to in your browser.
Follow Step 4: Test service federation by routing multicluster traffic to create a virtual destination, and configure an east-west route table for weighted, multi-cluster routing.

Next steps

If you haven't already, follow the other guides in the routing section to plan your routing table setup. For example, you might check out the prefix matching guide to decide how to match the incoming requests to your service paths, the redirect guide to set up any path or prefix rewrites, or the sub-table delegation guide to nest and sort multiple route tables.
Configure additional route settings, such as weighted routing to version subsets or adding and removing headers.