Multicluster federation and isolation with Bookinfo

After you install Gloo Mesh Enterprise, use the Bookinfo sample app and Gloo Mesh resources to test multitenancy, federation, and isolation across multiple clusters.

• Multitenancy: Gloo Mesh workspaces let you delegate management and policy decisions to your teams. You create workspaces, and select the appropriate namespaces and clusters for the workspaces. Note that in multicluster setups, a workspace can even include namespaces across clusters. Afterwards, the rest of your teams access only the namespaces in the clusters that they need.
• Federation: With Gloo Mesh, you can federate trust so that services in different clusters can communicate with each other. In this example, Gloo Mesh handles the certificates for you, but you can optionally bring your own. Federation lets you easily reuse policies and other resources across your multi-mesh environment.
• Isolation: To enforce a zero trust model from the start, you can enable isolation for services. Isolated services cannot communicate with services outside the mesh or in another workspace, and mTLS is used for communication across services. Teams must set up their workspaces to import and export the services that they need.

The following figure depicts the multi-mesh architecture created by this quick-start guide.

Step 1: Set up multitenancy by creating workspaces for your workloads

In v2 of the Gloo Mesh API, Gloo Mesh introduces a new concept, the Workspace custom resource. A workspace consists of one or more Kubernetes namespaces that are in one or more clusters. Think of a workspace as the boundary of your team's resources. To get started, create a workspace for each of your teams. Your teams might start with their apps in a couple Kubernetes namespaces in a single cluster. As your teams scale across namespaces and clusters, their workspaces scale with them. For more information, see the Multi-tenancy concept.

1. Create the bookinfo namespace in each workload cluster.

   kubectl create ns bookinfo --context $REMOTE_CONTEXT1
   kubectl create ns bookinfo --context $REMOTE_CONTEXT2
   

2. Create a bookinfo workspace that spans across both cluster-1 and cluster-2, and includes only the bookinfo namespaces in each cluster. Note that you must create the workspace resource in the gloo-mesh namespace of the management cluster. For more information about setting up workspaces, see Create a workspace.

   kubectl apply --context $MGMT_CONTEXT -f- <<EOF
   apiVersion: admin.gloo.solo.io/v2
   kind: Workspace
   metadata:
     name: bookinfo
     namespace: gloo-mesh
   spec:
     workloadClusters:
       - name: ${REMOTE_CLUSTER1}
         namespaces:
         - name: bookinfo
       - name: ${REMOTE_CLUSTER2}
         namespaces:
         - name: bookinfo
   EOF
   

3. Configure settings for the bookinfo workspace, which include:

   • Enabled federation: Services in different clusters can communicate with each other. For example, even though the services in the bookinfo workspace are in different clusters, they are able to communicate with each other. In subsequent steps, you create routes in a route table to forward traffic between Bookinfo services across clusters.
   • Enabled service isolation: Services are isolated and cannot communicate with services outside the mesh or in another workspace by default. mTLS is used for communication across services.
   • Exporting: Because service isolation is enabled, the resources in this workspace are exported to the istio-system workspace. This setting enables the ingress gateway in the istio-system workspace to use the resources in the bookinfo workspace.

   kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF
   apiVersion: admin.gloo.solo.io/v2
   kind: WorkspaceSettings
   metadata:
     name: bookinfo-settings
     namespace: bookinfo
   spec:
     exportTo:
     - workspaces:  
       - name: istio-system
     options:
       serviceIsolation:
         enabled: true
       federation:
         enabled: true
         serviceSelector:
           - {}
         hostSuffix: 'global'
   EOF
   

4. Create an istio-system workspace that spans across both cluster-1 and cluster-2, and includes only the istio-system namespaces in each cluster. Note that in production-level environments, you might deploy ingress and east-west gateways into their own namespaces, and create separate workspaces for them. For simplicity, this getting started guide includes the gateways into istio-system.

   kubectl apply --context $MGMT_CONTEXT -f- <<EOF
   apiVersion: admin.gloo.solo.io/v2
   kind: Workspace
   metadata:
     name: istio-system
     namespace: gloo-mesh
   spec:
     workloadClusters:
       - name: ${REMOTE_CLUSTER1}
         namespaces:
         - name: istio-system
       - name: ${REMOTE_CLUSTER2}
         namespaces:
         - name: istio-system
   EOF
   

5. Configure settings for the istio-system workspace. Resources are imported for use from the bookinfo workspace.

   kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF
   apiVersion: admin.gloo.solo.io/v2
   kind: WorkspaceSettings
   metadata:
     name: istio-system-settings
     namespace: istio-system
   spec:
     importFrom:
     - workspaces:
       - name: bookinfo
   EOF
   

Step 2: Deploy Bookinfo across clusters

To test out service federation and isolation, deploy different versions of the Bookinfo sample app to both of the workload clusters. cluster-1 runs the app with versions 1 and 2 of the reviews service (reviews-v1 and reviews-v2), and cluster-2 runs version 3 of the reviews service (reviews-v3).

1. OpenShift only: Create a NetworkAttachmentDefinition custom resource for the bookinfo project of each workload cluster, and elevate the permissions of the bookinfo service account to allow the Istio sidecars to make use of a user ID that is normally restricted by OpenShift. Use the commands in both tabs.

   • cluster-1
   • cluster-2

   
   cat <<EOF | oc --context $REMOTE_CONTEXT1 -n bookinfo create -f -
   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: istio-cni
   EOF
   oc --context $REMOTE_CONTEXT1 adm policy add-scc-to-group anyuid system:serviceaccounts:bookinfo
   

   
   cat <<EOF | oc --context $REMOTE_CONTEXT2 -n bookinfo create -f -
   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: istio-cni
   EOF
   oc --context $REMOTE_CONTEXT2 adm policy add-scc-to-group anyuid system:serviceaccounts:bookinfo
   

2. Export the Istio version that your cluster runs as an environment variable, such as 1.13.5 in the following example. You can check your Istio version by running istioctl version --context $REMOTE_CONTEXT1.

   export ISTIO_VERSION=1.13.5
   

3. Use the commands in both tabs to install Bookinfo with the reviews-v1 and reviews-v2 services in cluster-1, and the reviews-v3 service in cluster-2.

   • cluster-1
   • cluster-2

   
   # prepare the bookinfo namespace for Istio sidecar injection
   kubectl --context $REMOTE_CONTEXT1 label namespace bookinfo istio-injection=enabled
   # deploy bookinfo application components for all versions less than v3
   kubectl --context $REMOTE_CONTEXT1 -n bookinfo apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml -l 'app,version notin (v3)'
   # deploy all bookinfo service accounts
   kubectl --context $REMOTE_CONTEXT1 -n bookinfo apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml -l 'account'
   

   
   # prepare the bookinfo namespace for Istio sidecar injection
   kubectl --context $REMOTE_CONTEXT2 label namespace bookinfo istio-injection=enabled
   # deploy reviews and ratings services
   kubectl --context $REMOTE_CONTEXT2 -n bookinfo apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml -l 'service in (reviews)'
   # deploy reviews-v3
   kubectl --context $REMOTE_CONTEXT2 -n bookinfo apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml -l 'app in (reviews),version in (v3)'
   # deploy ratings
   kubectl --context $REMOTE_CONTEXT2 -n bookinfo apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml -l 'app in (ratings)'
   # deploy reviews and ratings service accounts
   kubectl --context $REMOTE_CONTEXT2 -n bookinfo apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml -l 'account in (reviews, ratings)'
   

4. Verify that the Bookinfo pods are have a status of Running in each cluster. If not, try Troubleshooting.

   kubectl --context $REMOTE_CONTEXT1 get pods -n bookinfo
   kubectl --context $REMOTE_CONTEXT2 get pods -n bookinfo
   

Step 3: Expose the ingress gateway

Use Gloo Mesh resources to set up routing with the Istio ingress gateway. The following example also makes use of the multitenancy features of Gloo Mesh workspaces. Your gateway delegates the route table from the istio-system to the bookinfo workspace. The workspaces’ import and export settings allow them to share these Gloo Mesh resources.

1. Create a virtual gateway for the ingress gateway (north-south traffic) in cluster-1. The VirtualGateway resource allows you to configure the istio-ingressgateway by using Gloo Mesh resources, and is translated to the Istio Gateway resource.

   kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF
   apiVersion: networking.gloo.solo.io/v2
   kind: VirtualGateway
   metadata:
     name: north-south-gw
     namespace: istio-system
   spec:
     workloads:
       - selector:
           labels:
             istio: ingressgateway
           cluster: ${REMOTE_CLUSTER1}
     listeners:
       - port:
           number: 80
         http: {}
   EOF
   

2. Create a route table for the productpage and reviews apps via the Istio ingress gateway in cluster-1. This resource allows you to define how incoming north-south requests to endpoints should be routed, and is translated to the Istio VirtualService resource. Note that this reviews route does not specify v1 or v2 of the app in cluster-1, nor includes v3 of the app in cluster-2.

   kubectl apply --context $REMOTE_CONTEXT1 -n bookinfo -f- <<EOF
   apiVersion: networking.gloo.solo.io/v2
   kind: RouteTable
   metadata:
     name: bookinfo-north-south
     namespace: bookinfo
     labels:
       workspace.solo.io/exported: 'true'
   spec:
     hosts:
       - '*'
     # Selects the virtual gateway you previously created
     virtualGateways:
       - name: north-south-gw
         namespace: istio-system
         cluster: ${REMOTE_CLUSTER1}
     http:
       # Route for the main productpage app
       - name: productpage
         matchers:
         - uri:
             prefix: /productpage
         - uri:
             prefix: /static
         forwardTo:
           destinations:
             - ref:
                 name: productpage
                 namespace: bookinfo
                 cluster: ${REMOTE_CLUSTER1}
               port:
                 number: 9080
       # Routes all /reviews requests to the reviews-v1 or reviews-v2 apps in cluster-1
       - name: reviews
         labels:
           route: reviews
         matchers:
         - uri:
             prefix: /reviews
         forwardTo:
           destinations:
             - ref:
                 name: reviews
                 namespace: bookinfo
                 cluster: ${REMOTE_CLUSTER1}
               port:
                 number: 9080
   EOF
   

3. Get the address of the Istio ingress gateway on cluster-1.

   • GKE: LoadBalancer IP address
   • EKS: LoadBalancer hostname

   
   export CLUSTER_1_INGRESS_ADDRESS=$(kubectl --context $REMOTE_CONTEXT1 get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   echo http://$CLUSTER_1_INGRESS_ADDRESS/productpage
   

   
   export CLUSTER_1_INGRESS_ADDRESS=$(kubectl --context $REMOTE_CONTEXT1 get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   echo http://$CLUSTER_1_INGRESS_ADDRESS/productpage
   

4. Navigate to http://$CLUSTER_1_INGRESS_ADDRESS/productpage in a web browser.

   open http://$CLUSTER_1_INGRESS_ADDRESS/productpage
   

5. Refresh the page a few times to see the black stars in the Book Reviews column appear and disappear. The presence of black stars represents reviews-v1 and the absence of black stars represents reviews-v2. Note that the styling of red stars from reviews-v3 is not shown because the services in cluster-1 do not currently communicate with the services in cluster-2.

Step 4: Test service federation by routing multicluster traffic

Next, to test whether your services are federated across clusters and service meshes, use Gloo Mesh to route traffic across the workload clusters. In order for the product-page service on cluster-1 to access reviews-v3 on cluster-2, you create a virtual destination that represents all versions of the reviews app across both clusters. Then, you create a route table to route from product-page to the virtual destination, and divert 75% of reviews traffic to the reviews-v3 service.

1. Create a virtual destination resource and define a unique hostname that in-mesh gateways can use to send reviews requests to. This virtual destination is configured to listen for incoming traffic on the internal-only, arbitrary hostname reviews.mesh.internal.com:8080. Note that this hosts value is different than the actual internal address that the reviews app can be reached by, because this host is an internal address that is used only by the gateways in your mesh.

   kubectl apply --context $REMOTE_CONTEXT1 -n bookinfo -f- <<EOF
   apiVersion: networking.gloo.solo.io/v2
   kind: VirtualDestination
   metadata:
     name: reviews-vd
     namespace: bookinfo
   spec:
     hosts:
     # Arbitrary, internal-only hostname assigned to the endpoint
     - reviews.mesh.internal.com
     ports:
     - number: 8080
       protocol: HTTP
       targetPort:
         number: 9080
     services:
       - labels:
           app: reviews
   EOF
   

2. Create a second route table that defines how east-west requests within your mesh from the productpage service to the reviews-vd virtual destination should be routed. Create 3 references for the virtual destination, and specify the subset version and a weight in each reference. When you apply this route table, requests from productpage to /reviews now route to one of the three reviews versions depending on their differently assigned weights.

   • For hosts, specify reviews.bookinfo.svc.cluster.local, which is the actual internal hostname that the reviews app listens on. The east-west gateway in your mesh does the work of taking requests made to the reviews.bookinfo.svc.cluster.local hostname and routing them to the reviews.mesh.internal.com virtual destination hostname that you specified in the previous step.
   • Create 3 references for the virtual destination, and specify the subset version and a weight in each reference. When you apply this route table, requests from productpage to /reviews now route to one of the three reviews versions depending on their differently assigned weights.

   kubectl apply --context $REMOTE_CONTEXT1 -n bookinfo -f- <<EOF
   apiVersion: networking.gloo.solo.io/v2
   kind: RouteTable
   metadata:
     name: bookinfo-east-west
     namespace: bookinfo
   spec:
     hosts:
       - 'reviews.bookinfo.svc.cluster.local'
     workloadSelectors:
       - selector:
           labels:
             app: productpage
     http:
       - name: reviews
         matchers:
         - uri:
             prefix: /reviews
         forwardTo:
           destinations:
             # Reference to the virtual destination that directs 15% of reviews traffic to reviews-v1 in cluster-1
             - ref:
                 name: reviews-vd
               kind: VIRTUAL_DESTINATION
               port:
                 number: 8080
               subset:
                 version: v1
               weight: 15
             # Reference to the virtual destination that directs 10% of reviews traffic to reviews-v2 in cluster-1
             - ref:
                 name: reviews-vd
               kind: VIRTUAL_DESTINATION
               port:
                 number: 8080
               subset:
                 version: v2
               weight: 10
             # Reference to the virtual destination that directs 75% of reviews traffic to reviews-v3 in cluster-2
             - ref:
                 name: reviews-vd
               kind: VIRTUAL_DESTINATION
               port:
                 number: 8080
               subset:
                 version: v3
               weight: 75
   EOF
   

3. In the http://$CLUSTER_1_INGRESS_ADDRESS/productpage page in your web browser, refresh the page a few times again. Now, the red stars for reviews-v3 are shown in the book reviews.

Bookinfo services in cluster-1 are now successfully accessing the Bookinfo services in cluster-2!

Step 5: Test service isolation by blocking access from services

In Step 1, you enabled service isolation for the Bookinfo workspace. When service isolation is enabled:

• Services in the mesh cannot communicate with services outside of the mesh.
• Communication between services within the mesh are permitted only when services are in the same workspace or when their workspaces have rules to import/export resources between them.

To verify that services within the workspace are isolated, deploy two httpbin demo apps to an httpbin namespace in cluster-1, which is not part of the bookinfo workspace. One version of the app is included in the Istio service mesh, and one version is not included in the mesh. Then, attempt to curl the Bookinfo services from the httpbin apps to test service isolation.

1. Create the httpbin namespace on cluster-1. Do not label this namespace for Istio injection.

   kubectl --context $REMOTE_CONTEXT1 create ns httpbin
   

2. Deploy the in-mesh app on cluster-1. These commands download the deployment file, manually inject the Istio sidecar to the deployment, and create the deplyment in your cluster.

   curl -0L https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/getting-started/2.1/in-mesh.yaml > in-mesh.yaml
   istioctl kube-inject -f in-mesh.yaml | kubectl apply --context $REMOTE_CONTEXT1 -n httpbin -f -
   

3. Deploy the not-in-mesh app on cluster-1. These commands download the deployment file and create the deplyment in your cluster, but no Istio sidecar is injected.

   curl -0L https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/getting-started/2.1/not-in-mesh.yaml > not-in-mesh.yaml
   kubectl apply --context $REMOTE_CONTEXT1 -n httpbin -f not-in-mesh.yaml
   

4. Verify that both app pods have a status of running.

   kubectl --context $REMOTE_CONTEXT1 -n httpbin get pods
   

5. Curl the reviews service from the in-mesh service. This curl returns a 403 response code because the sidecar of the reviews service blocks the request. Even though the services are both in the same Istio service mesh, they cannot communicate because they are not in the same workspace, and the httpbin namespace is not in a workspace with import/export rules for the Bookinfo workspace.

   pod=$(kubectl --context ${REMOTE_CONTEXT1} -n httpbin get pods -l app=in-mesh -o jsonpath='{.items[0].metadata.name}')
   kubectl --context ${REMOTE_CONTEXT1} -n httpbin debug -i ${pod} --image=curlimages/curl -- curl -s -o /dev/null -w "%{http_code}" http://reviews.bookinfo-backends.svc.cluster.local:9080/reviews/0
   

6. Curl the reviews service from the not-in-mesh service. This curl returns a 000 response code because communication cannot be established. Because the Bookinfo services are isolated, they cannot communicate with services outside of the same Istio service mesh.

   pod=$(kubectl --context ${REMOTE_CONTEXT1} -n httpbin get pods -l app=not-in-mesh -o jsonpath='{.items[0].metadata.name}')
   kubectl --context ${REMOTE_CONTEXT1} -n httpbin debug -i ${pod} --image=curlimages/curl -- curl -s -o /dev/null -w "%{http_code}" http://reviews.bookinfo-backends.svc.cluster.local:9080/reviews/0
   

7. Optional: For additional testing, you can change the Bookinfo workspace settings to disable service isolation. Then, when you repeat the commands in steps 5 - 6, the curl commands from both the in-mesh and not-in-mesh services now receive 200 response codes. Services within the same mesh and outside of the mesh can access the Bookinfo services because they are no longer isolated.

   kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF
   apiVersion: admin.gloo.solo.io/v2
   kind: WorkspaceSettings
   metadata:
     name: bookinfo-settings
     namespace: bookinfo
   spec:
     exportTo:
        - name: 'istio-system'
     options:
       serviceIsolation:
         enabled: false
       federation:
         enabled: true
         serviceSelector:
           - {}
         hostSuffix: 'global'
   EOF
   

Step 6: Launch the Gloo Mesh UI

The Gloo Mesh UI provides a single pane of glass through which you can observe the status of your service meshes, workloads, and services that run across all of your clusters. You can also view the policies that configure the behavior of your network.

1. Access the Gloo Mesh UI.

   meshctl dashboard --kubecontext $MGMT_CONTEXT
   

2. Click through the Workspace cards to view the configuration of your Bookinfo resources.

   • From the default collapsed view, you can see the number of clusters, namespaces, and gateways in each workspace.
   • When you expand the card, you see more information about the imported and exported resources, destinations, and policies.
   • Click MORE DETAILS to jump to the workspace details page for that workspace. Take a look at the Destinations and Routing tabs to review which resources are being exported. For example, in the details for the istio-system workspace, you can see the various Bookinfo destinations listed as imported to this workspace.

To learn more about what you can do with the UI, see the Gloo Mesh UI guides.

Next steps

Now that you have Gloo Mesh Enterprise up and running, check out some of the following resources to learn more about Gloo Mesh or try other Gloo Mesh features.

• Browse Gloo Mesh traffic control and security policies to try out some of Gloo Mesh Enterprise's features.
• Check out the setup guide for advanced installation and cluster registration options.
• Talk to an expert to get advice or build out a proof of concept.
• Join the #gloo-mesh channel in the Solo.io community slack.
• Try out one of the Gloo Mesh workshops.

Cleanup

If you no longer need this quick-start Gloo Mesh environment, you can deregister workload clusters, uninstall management components from the management cluster, and uninstall Istio resources from the workload clusters by following the steps in Uninstalling Gloo Mesh and Istio.