You can deploy Gloo Mesh Enterprise alongside Gloo Gateway to set up secure ingress routing for services in a service mesh that are spread across multiple clusters. To accomplish multicluster routing, you leverage Gloo Mesh Enterprise VirtualDestinations.

About Gloo Mesh Enterprise

Gloo Mesh Enterprise is a service mesh management plane that is based on hardened, open-source projects like Envoy and Istio. With Gloo Mesh, you can unify the configuration, operation, and visibility of service-to-service connectivity across your distributed applications. These apps can run in different virtual machines (VMs) or Kubernetes clusters on premises or in various cloud providers, and even in different service meshes.

About VirtualDestinations

A VirtualDestination is a traffic management concept in Gloo Mesh Enterprise that allows you to define unique internal hostnames for services that are spread across a multicluster service mesh. Without VirtualDestinations, you must update the IP addresses of your services every time the IP address changes, such as when you update your app or deploy a new version. Because a VirtualDestination uses labels to select the backing services, it can automatically detect IP address changes and update them for you.

To learn more about VirtualDestinations, see the VirtualDestinations conceptual overview and multicluster routing guide in the Gloo Mesh Enterprise documentation.

About this guide

In this guide, you explore how to use Gloo Mesh Enterprise and Gloo Gateway together to route traffic to Bookinfo services that are spread across clusters. Bookinfo is a sample microservices-based app that is provided by Istio and composed of 4 different microservices that interact with each other. The app is commonly used to to demonstrate Istio’s service mesh features and capabilities.

To accomplish multicluster routing with Gloo Gateway and Gloo Mesh Enterprise, you leverage VirtualDestinations. You can use VirtualDestinations in multiple ways in Gloo Gateway. This guide provides steps to accomplish the following tasks:

  • Expose the product page app on a gateway, which is part of a service mesh that Gloo Mesh Enterprise manages. The product page app uses a VirtualDestination to round robin requests to reviews app instances that are spread across multiple clusters. The VirtualDestination is not exposed on the gateway directly, but rather used within the service mesh only.
  • Expose the reviews VirtualDestination on the gateway directly. This way, you can explore how to use a unique internal hostname to round robin traffic to reviews app instances that are spread across clusters.

Step 1: Install Gloo Mesh Enterprise

  1. Install Gloo Mesh Enterprise by following the multicluster getting started tutorial in the Gloo Mesh Enterprise documentation. This guide creates a three-cluster setup with one management cluster that runs the Gloo Mesh Enterprise control plane and two workload clusters that run the data plane. You also install a Solo distribution of Istio in both of your workload clusters by using the IstioLifecycleManager.
  2. Deploy the sample Bookinfo app. You use this app to demonstrate traffic routing in a multicluster service mesh. You can optionally install other sample apps, such as httpbin or helloworld. However, these apps are not used in this guide.
  3. Set up multicluster routing for the reviews app by using a VirtualDestination and RouteTable resource. Make sure that the productpage app can route traffic to all versions of the reviews app before you continue with installing Gloo Gateway.

Step 2: Install Gloo Gateway

After you successfully installed Gloo Mesh Enterprise and confirmed that you can route traffic from the product page app to all versions of the reviews app, you can now continue to install Gloo Gateway. Gloo Gateway must be installed in a cluster that also has Istio installed. You can choose to install Gloo Gateway in one or each of your workload clusters.

This guide assumes that you want to install Gloo Gateway in workload cluster 1.

  1. Set your Gloo Gateway license key as an environment variable. If you do not have one, contact an account representative.

      export GLOO_GATEWAY_LICENSE_KEY=<license-key>

  2. Install the custom resources of the Kubernetes Gateway API. To use VirtualDestinations with Gloo Gateway, you must install the experimental channel of version 1.2.1.

      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/experimental-install.yaml --context $REMOTE_CONTEXT1

    Example output:

      customresourcedefinition.apiextensions.k8s.io/backendlbpolicies.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/backendtlspolicies.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/tcproutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/tlsroutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/udproutes.gateway.networking.k8s.io created

  3. Get the name of the istiod service. Depending on how you set up Istio, you might see a revisionless service name (istiod) or a service name with a revision, such as istiod-1-22.

      kubectl get services -n istio-system --context $REMOTE_CONTEXT1

    Example output:

      NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                 AGE
istiod-1-22   ClusterIP   34.118.238.13   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   3d2h

  4. Derive the Kubernetes service address for your istiod deployment. The service address uses the format <service-name>.<namespace>.svc:15012. For example, if your service name is istiod-1-22, the full service address is istiod-1-22.istio-system.svc:15012.

  5. Install Gloo Gateway and enable the Istio integration.

    1. Add the Helm repository for Gloo Gateway Enterprise Edition.

        helm repo add glooe https://storage.googleapis.com/gloo-ee-helm
helm repo update

    2. Install Gloo Gateway Enterprise Edition. This command creates the gloo-system namespace and installs the Gloo Gateway control plane into it. Replace the following values:

      • istioDiscoveryAddress: Set <istiod-service-address> to the istiod service address that you retrieved earlier.
      • istioMetaClusterId: The trustDomain that was set in the IstioOperator for istiod. If you followed the getting started tutorial in the Gloo Mesh Enterprise documentation, you must set this value to the name of your workload cluster.
      • istioMetaMeshId: The meshID that was set in the IstioOperator for istiod. If you followed the getting started tutorial in the Gloo Mesh Enterprise documentation, you must set this value to the name of your workload cluster.
        helm install -n gloo-system gloo-gateway glooe/gloo-ee \
--kube-context $REMOTE_CONTEXT1 \
--create-namespace \
--version 1.19.0-beta2 \
--set-string license_key=$GLOO_GATEWAY_LICENSE_KEY \
-f -<<EOF
gloo:
  discovery:
    enabled: false
  gatewayProxies:
    gatewayProxy:
      disabled: true
  kubeGateway:
    enabled: true
    gatewayParameters:
      glooGateway:
        istio:
          istioProxyContainer:
            istioDiscoveryAddress: <istiod-service-address>
            istioMetaClusterId: <trustDomain>
            istioMetaMeshId: <meshID>
  gloo:
    disableLeaderElection: true
gloo-fed:
  enabled: false
  glooFedApiserver:
    enable: false
grafana:
  defaultInstallationEnabled: false
observability:
  enabled: false
prometheus:
  enabled: false
global:
  istioSDS:
    enabled: true
  istioIntegration:
    enabled: true
    enableAutoMtls: true
EOF

      Example output:

        NAME: gloo-gateway
LAST DEPLOYED: Thu Dec 9 11:50:39 2024
NAMESPACE: gloo-system
STATUS: deployed
REVISION: 2
TEST SUITE: None

  6. Verify that the Gloo Gateway control plane is up and running.

      kubectl get pods -n gloo-system --context $REMOTE_CONTEXT1

    Example output:

      NAME                                READY   STATUS      RESTARTS   AGE
extauth-64458459c8-9q9kq            1/1     Running     0          26s
gateway-certgen-qpv5q               0/1     Completed   0          31s
gloo-68846f8459-nbzxp               1/1     Running     0          26s
gloo-resource-rollout-check-xskxt   0/1     Completed   0          25s
gloo-resource-rollout-n9kvf         0/1     Completed   0          26s
rate-limit-5d8f859465-qhgsg         1/1     Running     0          26s
redis-9f4f98445-5fh69               1/1     Running     0          26s

  7. Verify that the gloo-gateway GatewayClass is created. You can optionally take a look at how the gateway class is configured by adding the -o yaml option to your command.

      kubectl get gatewayclass gloo-gateway --context $REMOTE_CONTEXT1

    Example output:

      NAME           CONTROLLER             ACCEPTED   AGE
gloo-gateway   solo.io/gloo-gateway   True       56s

Step 3: Create a gateway proxy

  1. Create a gateway resource and configure an HTTP listener. The following gateway can serve HTTP resources from all namespaces.

      kubectl apply --context $REMOTE_CONTEXT1 -n gloo-system -f- <<EOF
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: http
spec:
  gatewayClassName: gloo-gateway
  listeners:
  - protocol: HTTP
    port: 8080
    name: http
    allowedRoutes:
      namespaces:
        from: All
EOF

  2. Verify that the gateway is created successfully. You can also review the external address that is assigned to the gateway. Note that depending on your environment it might take a few minutes for the load balancer service to be assigned an external address.

      kubectl get gateway http -n gloo-system --context $REMOTE_CONTEXT1

    Example output:

      NAME   CLASS          ADDRESS                                                                  PROGRAMMED   AGE
http   gloo-gateway   a3a6c06e2f4154185bf3f8af46abf22e-139567718.us-east-2.elb.amazonaws.com   True         93s

  3. Check the gateway proxy pod that is created for you. Make sure that you see three containers in your pod (istio-proxy, sds, and gloo-gateway).

      kubectl get pod -n gloo-system --context $REMOTE_CONTEXT1 | grep gloo-proxy-http

    Example output:

      gloo-proxy-http-bc64bfbd4-qkgvh   3/3     Running   0          124m

  4. Get the address of your gateway proxy and save it in an environment variable.

      export INGRESS_GW_ADDRESS=$(kubectl get svc --context $REMOTE_CONTEXT1 -n gloo-system gloo-proxy-http -o=jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS

    Port-forward the gloo-proxy-http pod on port 8080.

      kubectl port-forward deployment/gloo-proxy-http -n gloo-system --context $REMOTE_CONTEXT1 8080:8080

Step 4: Expose the product page app

  1. Set up an HTTPRoute that exposes a route to the productpage app.

      kubectl apply --context $REMOTE_CONTEXT1 -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: productpage
  namespace: bookinfo
spec:
  parentRefs:
    - name: http
      namespace: gloo-system
  rules:
    - matches:
      - path:
          type: Exact
          value: /productpage
      - path: 
          type: PathPrefix
          value: /static
      backendRefs: 
        - name: productpage
          port: 9080
          namespace: bookinfo
EOF

  2. Open the product page in your web browser.

      open http://$INGRESS_GW_ADDRESS:8080/productpage

  3. Refresh the page a few times. Because you have the Gloo Mesh Enterprise RouteTable in place, the product page round robins through all the reviews instances in workload clusters 1 and 2 by using the VirtualDestination. Make sure that you see all three versions of the reviews app: black stars, no stars, and red stars.

Step 5: Route to a VirtualDestination

In this step, you expose the VirtualDestination for the reviews app on your gateway proxy directly.

  1. Create an HTTPRoute for the reviews app that references the VirtualDestination in your backendRefs. In order for the HTTPRoute to match the VirtualDestination, you use the hostname that your VirtualDestination is exposed on. Note that this hostname must match the hostname that is defined in the ServiceEntry resource that is automatically created by Gloo Mesh Enterprise during the translation of your VirtualDestination.

      kubectl apply --context $REMOTE_CONTEXT1 -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: reviews
  namespace: gloo-system
spec:
  parentRefs:
    - name: http
      namespace: gloo-system
  rules:
    - matches:
      - path: 
          type: PathPrefix
          value: /reviews
      backendRefs: 
        - name: reviews.mesh.internal.com 
          port: 9080
          kind: Hostname
          group: networking.istio.io
EOF

  2. Send multiple requests to the reviews app directly. Make sure that you get back a response from all the reviews app instances.

      for i in {1..5}; do curl -v http://$INGRESS_GW_ADDRESS:8080/reviews/0 ; done 
  
      for i in {1..5};, do curl -i localhost:8080/reviews/0; done

    Example output:

      ...
{"id": "0","podname": "reviews-v1-7cb6f8bdb9-dwxfk","clustername": "null","reviews": [{  "reviewer": "Reviewer1",  "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!"},{  "reviewer": "Reviewer2",  "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare."}]}* 
...
{"id": "0","podname": "reviews-v2-778547d5c9-qqgv4","clustername": "null","reviews": [{  "reviewer": "Reviewer1",  "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!", "rating": {"stars": 5, "color": "black"}},{  "reviewer": "Reviewer2",  "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare.", "rating": {"stars": 4, "color": "black"}}]}*  
...
{"id": "0","podname": "reviews-v3-66d547bb6f-fzzqt","clustername": "null","reviews": [{  "reviewer": "Reviewer1",  "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!", "rating": {"stars": 5, "color": "red"}},{  "reviewer": "Reviewer2",  "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare.", "rating": {"stars": 4, "color": "red"}}]}%

Great job! You successfully exposed a VirtualDestination on your gateway and routed traffic to app instances that are spread across clusters.

Cleanup

You can optionally remove the resources that you created.

  1. Follow the Uninstall guide in the Gloo Mesh Enterprise documentation to remove Gloo Mesh Enterprise.

  2. Follow the upgrade guide to upgrade your Gloo Gateway Helm installation values and disable the Istio integration.

  3. Remove the HTTPRoutes for the product page and reviews app.

      kubectl delete httproute productpage reviews -n bookinfo