Tutorial: Federate clusters and isolate workloads for multitenancy

Use the Bookinfo sample app and Gloo resources to test multitenancy, federation, and isolation across multiple clusters.

The following figure depicts the multi-mesh architecture created by this guide.

Figure of multicluster Bookinfo traffic in a Gloo Mesh quick-start architecture.

Before you begin

  1. Install Gloo Mesh in a multicluster environment.
  2. Install Bookinfo and other sample apps.
  3. Set up routing for the sample apps.

Set up multitenancy by creating workspaces for your workloads

Gloo introduces a new concept for Kubernetes-based multitenancy, the Workspace custom resource. A workspace consists of one or more Kubernetes namespaces that are in one or more clusters. Think of a workspace as the boundary of your team's resources. To get started, you can create a workspace for each of your teams. Your teams might start with their apps in a couple Kubernetes namespaces in a single cluster. As your teams scale across namespaces and clusters, their workspaces scale with them. Note that for simplicity, you create all your Gloo custom resources only in the management cluster. For more information, see the Multitenancy concept.

  1. Create a bookinfo workspace that spans across all your clusters, and includes only the bookinfo namespaces in each cluster. Note that you must create the workspace resource in the gloo-mesh namespace of the management cluster. For more information about setting up workspaces, see Create a workspace.

    kubectl apply --context $MGMT_CONTEXT -f- <<EOF
    apiVersion: admin.gloo.solo.io/v2
    kind: Workspace
    metadata:
      name: bookinfo
      namespace: gloo-mesh
    spec:
      workloadClusters:
        - name: '*'
          namespaces:
          - name: bookinfo
    EOF
    
  2. Configure settings for the bookinfo workspace, which include:

    • Enabled federation: Services in different clusters can communicate with each other. For example, even though the services in the bookinfo workspace are in different clusters, they are able to communicate with each other. In subsequent steps, you create routes in a route table to forward traffic between Bookinfo services across clusters.
    • Enabled service isolation: Services are isolated and cannot communicate with services outside the mesh or in another workspace by default. mTLS is used for communication across services.
    • Exporting: Because service isolation is enabled, the resources in this workspace are exported to the istio-system workspace. This setting enables the ingress gateway in the istio-system workspace to use the resources in the bookinfo workspace.
    kubectl apply --context $MGMT_CONTEXT -f- <<EOF
    apiVersion: admin.gloo.solo.io/v2
    kind: WorkspaceSettings
    metadata:
      name: bookinfo-settings
      namespace: bookinfo
    spec:
      exportTo:
      - workspaces:  
        - name: istio-system
      options:
        serviceIsolation:
          enabled: true
        federation:
          enabled: false
          serviceSelector:
            - {}
          hostSuffix: 'global'
    EOF
    
  3. Create an istio-system workspace that spans across clusters, and includes the istio-system and gloo-mesh-gateways namespaces in each cluster. This ensures that the istiod control plane components as well as the gateways are included in the same workspace.

    kubectl apply --context $MGMT_CONTEXT -f- <<EOF
    apiVersion: admin.gloo.solo.io/v2
    kind: Workspace
    metadata:
      name: istio-system
      namespace: gloo-mesh
    spec:
      workloadClusters:
        - name: '*'
          namespaces:
          - name: istio-system
          - name: gloo-mesh-gateways
    EOF
    
  4. Configure settings for the istio-system workspace. Resources are imported for use from the bookinfo workspace.

    kubectl apply --context $MGMT_CONTEXT -f- <<EOF
    apiVersion: admin.gloo.solo.io/v2
    kind: WorkspaceSettings
    metadata:
      name: istio-system-settings
      namespace: istio-system
    spec:
      importFrom:
      - workspaces:
        - name: bookinfo
    EOF
    
  5. Modify the default workspace that you created in the quick-start guide to become a management-only workspace.

    kubectl apply --context $MGMT_CONTEXT -f- <<EOF
    apiVersion: admin.gloo.solo.io/v2
    kind: Workspace
    metadata:
      name: $MGMT_CLUSTER
      namespace: gloo-mesh
    spec:
      workloadClusters:
        - name: '$MGMT_CLUSTER'
          namespaces:
            - name: 'gloo-mesh'
    EOF
    

As you create more workspaces, you can create global workspace settings in the management cluster that apply by default to each workspace in your Gloo Mesh environment. For more information, see Configure workspace settings.

Test service isolation by blocking access from services

In the previous section, you enabled service isolation for the Bookinfo workspace. When service isolation is enabled:

To verify that services within the workspace are isolated, deploy two demo apps to an isolation-test namespace in cluster1, which is not part of the bookinfo workspace. One version of the app is included in the Istio service mesh, and one version is not included in the mesh. Then, attempt to curl the Bookinfo services from the demo apps to test service isolation.

  1. Create the isolation-test namespace on cluster1. Do not label this namespace for Istio injection.

    kubectl create ns isolation-test --context $REMOTE_CONTEXT1
    
  2. Create an app named in-mesh that is part of the Istio service mesh.

    1. Get the Istio revision that your istiod control planes run, such as 1-18-3.
      kubectl get pod --context $REMOTE_CONTEXT1 -L app=istiod -n istio-system -o jsonpath='{.items[0].metadata.labels.istio\.io/rev}'
      
    2. Download the in-mesh app YAML file.
      curl -0L https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/policy-demo/httpbin-in-mesh.yaml > in-mesh.yaml
      open in-mesh.yaml
      
    3. In the Deployment, replace the value of spec.template.metadata.labels.istio.io/rev with your Istio revision, and save the file. This label ensures that an Istio sidecar is injected.
    4. Create the app in cluster1.
      kubectl-n isolation-test apply -f in-mesh.yaml --context $REMOTE_CONTEXT1 
      
  3. Create another app named not-in-mesh that is not part of the Istio service mesh.

    kubectl -n isolation-test apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/policy-demo/httpbin-not-in-mesh.yaml --context $REMOTE_CONTEXT1
    
  4. Verify that the app pods have a status of running.

    kubectl -n isolation-test get pods --context $REMOTE_CONTEXT1
    
  5. Curl the reviews service from the in-mesh service. This curl returns a 403 response code because the sidecar of the reviews service blocks the request. Even though the services are both in the same Istio service mesh, they cannot communicate because they are not in the same workspace, and the isolation-test namespace is not in a workspace with import/export rules for the Bookinfo workspace.

    pod=$(kubectl --context ${REMOTE_CONTEXT1} -n isolation-test get pods -l app=in-mesh -o jsonpath='{.items[0].metadata.name}')
    kubectl --context ${REMOTE_CONTEXT1} -n isolation-test debug -i ${pod} --image=curlimages/curl -- curl -s -o /dev/null -w "%{http_code}" http://reviews.bookinfo.svc.cluster.local:9080/reviews/0
    
  6. Curl the reviews service from the not-in-mesh service. This curl returns a 000 response code because communication cannot be established. Because the Bookinfo services are isolated, they cannot communicate with services outside of the same Istio service mesh.

    pod=$(kubectl --context ${REMOTE_CONTEXT1} -n isolation-test get pods -l app=not-in-mesh -o jsonpath='{.items[0].metadata.name}')
    kubectl --context ${REMOTE_CONTEXT1} -n isolation-test debug -i ${pod} --image=curlimages/curl -- curl -s -o /dev/null -w "%{http_code}" http://reviews.bookinfo.svc.cluster.local:9080/reviews/0
    
  7. Optional: For additional testing, you can change the Bookinfo workspace settings to disable service isolation. Then, when you repeat the commands in steps 5 - 6, the curl commands from both the in-mesh and not-in-mesh services now receive 200 response codes. Services within the same mesh and outside of the mesh can access the Bookinfo services because they are no longer isolated.

    kubectl apply --context $MGMT_CONTEXT -f- <<EOF
    apiVersion: admin.gloo.solo.io/v2
    kind: WorkspaceSettings
    metadata:
      name: bookinfo-settings
      namespace: bookinfo
    spec:
      exportTo:
      - workspaces:
         - name: 'istio-system'
      options:
        serviceIsolation:
          enabled: false
        federation:
          enabled: false
          serviceSelector:
            - {}
          hostSuffix: 'global'
    EOF
    

Next steps

Now that you have Gloo Mesh Enterprise up and running, check out some of the following resources to learn more about Gloo Mesh or try other Gloo Mesh features.

Cleanup

  1. Remove the demo app components.

    kubectl -n isolation-test delete -f in-mesh.yaml --context $REMOTE_CONTEXT1
    kubectl -n isolation-test delete -f https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/policy-demo/httpbin-not-in-mesh.yaml --context $REMOTE_CONTEXT1
    
  2. Delete the isolation-test namespace.

    kubectl delete ns isolation-test --context $REMOTE_CONTEXT1
    
  3. To revert your workspace setup to the single default workspace:

    1. Modify the default workspace to select all clusters and namespaces again.
      kubectl apply --context $MGMT_CONTEXT -f- <<EOF
      apiVersion: admin.gloo.solo.io/v2
      kind: Workspace
      metadata:
        name: $MGMT_CLUSTER
        namespace: gloo-mesh
      spec:
        workloadClusters:
          - name: '*'
            namespaces:
              - name: '*'
      EOF
      
    2. Delete the bookinfo and istio-system workspaces and workspace settings.
      kubectl delete WorkspaceSettings istio-system-settings --context $MGMT_CONTEXT
      kubectl delete Workspace istio-system --context $MGMT_CONTEXT
      kubectl delete WorkspaceSettings bookinfo-settings --context $MGMT_CONTEXT
      kubectl delete Workspace bookinfo --context $MGMT_CONTEXT
      
  4. If you no longer need this quick-start Gloo Mesh environment, you can deregister workload clusters, uninstall control plane components from the management cluster, and uninstall Istio resources from the workload clusters by following the steps in Uninstalling Gloo Mesh and Istio.