Navigation :

Set up multitenancy with workspaces

Gloo introduces a new concept for Kubernetes-based multitenancy, the Workspace custom resource. A workspace consists of one or more Kubernetes namespaces that are in one or more clusters. Think of a workspace as the boundary of your team's resources. To get started, you can create a workspace for each of your teams. Your teams might start with their apps in a couple Kubernetes namespaces in a single cluster. As your teams scale across namespaces and clusters, their workspaces scale with them. For more information, see the Workspaces concept.

Before you begin

Install Gloo Platform in your management cluster and register workload clusters. Note that you do not need to create a workspace as part of this setup as this guide walks you through how to configure the workspace.
If you are new to workspaces, review the following resources to understand more about the concepts.
- Workspaces concept, including how workspaces function as resource boundaries and a person-driven setup flow.
- API reference
Choose from the following options to create your workspace-related configuration files:
- Build config files with the Gloo UI: Quickly get started by building a simple config file for your workspace setup.
- Write your own config files: Start building your workspace setup from scratch. This detailed approach helps you prepare more complex setups such as for multiple tenants in production environments.

Build config files with the Gloo UI

The Gloo UI builder walks you through all the details to build the config files that you need for your workspace setup. The builder makes it easy to select resources that already exist in your environment, such as clusters and namespaces. Then, you can download and apply the config files, such as in your GitOps pipeline.

Launch the Gloo UI.
From the Overview page in the Workspaces pane, click Add Workspace +. The Workspace Builder modal opens.
Fill out the General Settings page as follows.
1. Workspace Name: Enter a unique name, which might be the same as your team's name, such as web-team.
2. Add Labels: Optionally add Kubernetes labels to your workspace. Later, other workspaces can use this label to import or export resources to your workspace. You might want to set up a naming convention for all of your workspaces. For example, you might use gloo.solo.io/global for workspaces that are available to any other workspace.
3. Additional Options: Enable certain settings at the workspace level.
  - Enable Federation: Decide if you want to set up federation for multicluster routing across your workspace. At the workspace level, federation creates ungrouped Istio service entries in each cluster so that services are accessible across clusters. For single clusters or to improve performance at scale in large environments, disable federation and use virtual destinations instead. For more information, see Federation.
  - Enable Service Isolation: Decide if you want to set up service isolation for your workspace. When service isolation is enabled, only services within the same workspace and services that are imported to the workspace can communicate with each other by default. Communication to services outside the workspace is blocked. However, to build a zero-trust architecture, you might want to disable service isolation and instead set up more fine-grained access control with access policies and trim proxy config policies. For more information, see Service isolation.
  - Enable Trim Proxy Config: Decide if you want to trim the Istio sidecar proxies so that they cannot talk to all destinations in the workspace by default. Then, you can add back in proxy config for only the destinations that you want the sidecar proxies to talk to. For more fine-grained control, you can use a TrimProxyConfigPolicy instead. For more information, see Proxy trimming.
4. Click Next Step.
Fill out the Inclusions page as follows.
1. From the Add Selections tab, choose the clusters and namespaces that you want to include in the workspace.
  - Choose Clusters: Select the clusters that you want the workspace to include. A workspace can have many clusters. Also, the same cluster might belong to several workspaces.
  - Choose Namespaces: Select the Kubernetes namespaces in the cluster that you want the workspace to include. Unlike a cluster, a namespace can belong to only one workspace.
  - Advanced: Instead of selecting individual clusters or namespaces by name, you can use labels or regex. This advanced approach allows you to treat clusters and namespaces “like cattle.” Gloo automatically adds or removes clusters and namespaces that meet the label or regex to your workspace. But be careful! This approach can also lead to clusters or namespaces being unintentionally added to your Gloo setup in the future.
Under Preview Selection, review the cluster and namespace selections that you just made.
To add the cluster and namespace to your workspace config, click Add Inclusion Rule. You see a Success message, and the inclusion rule is now in the Review Selections tab.
Optional: Repeat the previous steps to add more clusters and namespaces.
Click the Review Selections tab and review your inclusion rules.
When you are satisfied with your inclusion rules, click Next Step.
Fill out the Imports page as follows. Importing lets your workspace share resources from other teams, such as virtual destinations or route tables. The other workspace must also export to your workspace for the sharing to work. For more information, see Import and export resources across workspaces.
1. If you do not want to import other workspaces, toggle Skip Imports.
2. From the Import Declarations tab in the Select Workspaces list, select from available workspaces in your Gloo setup. The workspaces must already exist to show up in the list.
3. In the Filter By list, filter the resources that you want to import. For example, you might want to import only Kubernetes services that have a particular label and are in a particular cluster. Resources must meet all the filter conditions that you configure.
4. Under Preview Selection, review the workspaces and filtered resources that you want to import.
5. Click Add Import Rule. You see a Success message, and the import rule is now in the Review Import Rules tab.
6. Optional: Repeat the previous steps to import more resources from other workspaces.
7. Click the Review Import Rules tab and review your import rules.
8. When you are satisfied with your import rules, click Next Step.
Fill out the Exports page as follows. Exporting lets your workspace share resources with other teams, such as your virtual destinations or route tables. The other workspace must also import from your workspace for the sharing to work. For more information, see Import and export resources across workspaces.
1. If you do not want to export to other workspaces, toggle Skip Exports.
2. From the Export Declarations tab in the Select Workspaces list, select from available workspaces in your Gloo setup. The workspaces must already exist to show up in the list.
3. In the Filter By list, filter the resources that you want to export. For example, you might want to export only Kubernetes services that have a particular label and are in a particular cluster. Resources must meet all the filter conditions that you configure.
4. In the Preview Selection, review the workspaces and filtered resources that you want to export.
5. Click Add Export Rule. You see a Success message, and the export rule is now in the Review Export Rules tab.
6. Optional: Repeat the previous steps to export more resources to other workspaces.
7. Click the Review Export Rules tab and review your export rules.
8. When you are satisfied with your export rules, click Next Step.
From the YAML Config page, get your workspace configuration file. You can choose to download or copy the config file.
The builder helps makes sure that you filled out your workspace details as best as possible. If you did not complete the previous steps, you are prompted to return to those pages. If you have a configuration error, such as a nonexistent selector, you might see an error message to help you reconfigure accordingly.
Optional: Add more advanced settings to your config files. For more information, refer to the API docs for Workspace and WorkspaceSettings.
Create your workspace and workspace settings by applying the config files to your cluster. For example, you might add the files to your GitOps pipeline, or manually with the kubectl apply -f <workspace-config-file>.yaml command.

Good job! You can review your workspace details from the Overview page in the Gloo UI. To confirm that importing and exporting work, see verify your workspace setup.

Write your own config files

Review the following steps to build your workspace setup from scratch. You create several resources, including workspaces and workspace settings for each team, as well as global workspace settings that apply across teams. This detailed approach helps you prepare more complex setups such as for multiple tenants in production environments.

Complete the prerequisite steps
Create workspaces
Configure workspace settings

Step 1: Complete the prerequisite steps

Install Gloo Platform in your environment and optionally familiarize yourself with workspace concepts.

Set the names of your clusters from your infrastructure provider.

export MGMT_CLUSTER=<management_cluster_name>
export REMOTE_CLUSTER=<remote_cluster_name>

Save the kubeconfig contexts for your clusters. Run kubectl config get-contexts, look for your cluster in the CLUSTER column, and get the context name in the NAME column. Note: Do not use context names with underscores. The context name is used as a SAN specification in the generated certificate that connects workload clusters to the management cluster, and underscores in SAN are not FQDN compliant. You can rename a context by running kubectl config rename-context "<oldcontext>" <newcontext>.
```
export MGMT_CONTEXT=<management-cluster-context>
export REMOTE_CONTEXT=<remote-cluster-context>
```

Step 2: Create workspaces

As a platform administrator, you can create a workspace for each of your teams.

Make sure to include all your workloads that you want to manage network traffic for in a workspace, including the ingress gateway proxies. You can organize your workloads into however many workspaces that you want by creating separate workspaces for different cluster and namespace combinations.

Pick a name for the workspace. This name might be the same as your team's name, such as web-team, and must be unique.
In the management cluster, create a namespace for each workspace. Later, each team can create all their Gloo custom resources in this namespace. This way, each team's resources are kept in their own workspaces, but all of the Gloo configuration is kept centrally located in the management cluster. For example, you might create the following namespaces for three teams: ops-team, web-team, and backend-apis-team.
```
kubectl create namespace --context $MGMT_CONTEXT ops-team
kubectl create namespace --context $MGMT_CONTEXT web-team
kubectl create namespace --context $MGMT_CONTEXT backend-apis-team
```
Decide which clusters you want the workspace to include. A workspace can have many clusters. Also, the same cluster might belong to several workspaces. Instead of selecting individual clusters by name, use labels. Labels allow you to treat clusters “like cattle.” Gloo adds or removes clusters with the same label to your workspace. Workspace labels must match the KubernetesCluster labels on the management cluster. You can check those labels with the following command.
```
kubectl get kubernetescluster -n gloo-mesh --context $MGMT_CONTEXT -o=jsonpath="{.items[*]['metadata.name', 'metadata.labels']}"
```
Decide which Kubernetes namespaces across clusters you want the workspace to include. Unlike a cluster, a namespace can belong to only one workspace. To list existing namespaces, you can use the following command.
```
kubectl get ns --context $MGMT_CONTEXT && kubectl get ns --context $REMOTE_CONTEXT
```
Come up with a label for your workspace. Later, other workspaces can use this label to import or export to your workspace. You might want to set up a naming convention for all of your workspaces. For example, you might use gloo.solo.io/global for workspaces that are available to any other workspace.

Create the workspace with the information that you just gathered. Note that you must create the workspace resource in the gloo-mesh namespace of the management cluster. For an example of three workspaces for the Ops, Web, and Backend API teams, see Example workspace YAML configuration.

Ops team
Web team
Backend APIs team


kubectl apply --context $MGMT_CONTEXT -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: Workspace
metadata:
  name: ops-team
  namespace: gloo-mesh
  labels:
    gloo.solo.io/team: 'ops'
spec:
  workloadClusters:
  - name: '$MGMT_CLUSTER'
    namespaces:
    - name: ops-team
  - name: '*'
    namespaces:
    - name: istio-gateways
    - name: gloo-mesh-addons
EOF


kubectl apply --context $MGMT_CONTEXT -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: Workspace
metadata:
  name: web-team
  namespace: gloo-mesh
  labels:
    gloo.solo.io/team: 'web'
spec:
  workloadClusters:
  - name: '$MGMT_CLUSTER'
    namespaces:
    - name: web-team
  - name: '*'
    namespaces:
    - name: web-ui
EOF


kubectl apply --context $MGMT_CONTEXT -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: Workspace
metadata:
  name: backend-apis-team
  namespace: gloo-mesh
  labels:
    gloo.solo.io/team: 'backend-apis'
spec:
  workloadClusters:
  - name: '$MGMT_CLUSTER'
    namespaces:
    - name: backend-apis-team
  - name: '*'
    namespaces:
    - name: backend-apis
EOF

Modify any workspaces that might have conflicting namespaces, such as the default workspace, because namespaces can belong to only one workspace.
1. Review existing workspaces to note which namespaces are already part of a workspace. For example, when you install Gloo Platform in a single cluster, a default workspace and workspace settings are created automatically for you. However, because this default workspace is used for demonstration purposes, it does not isolate any resources, and instead allows all Kubernetes and Gloo Platform resources in the workspace.
```
kubectl get workspaces -n gloo-mesh --context $MGMT_CONTEXT -o=jsonpath="{.items[*]['metadata.name', 'spec.workloadClusters']}" 
```
  The following example output shows the default workspace called cluster1 that selects all clusters ("name":"*") and namespaces ("namespaces":[{"name":"*"}]) in your Gloo Platform setup.
```
cluster1 [{"name":"*","namespaces":[{"name":"*"}]}]
```
2. Modify the default workspace and any other workspaces with conflicting namespaces. For example, you might modify the default workspace so that it selects only the gloo-mesh namespace, instead of all namespaces.
```
kubectl edit workspace $CLUSTER_NAME
```
```
apiVersion: admin.gloo.solo.io/v2
kind: Workspace
metadata:
  name: gloo
  namespace: gloo-mesh
spec:
  workloadClusters:
    - name: '*'
      namespaces:
        - name: 'gloo-mesh'
         
```
Optionally, you can configure global workspace settings. These settings apply by default to each workspace in your Gloo environment. Later, the app owner for each workspace can override the defaults in their own workspace settings. For more information, see Workspace configuration.
You must create the global workspace settings resource in the gloo-mesh namespace of the management cluster, and name the resource global. If you have only one global workspace, then you can have only one workspace settings. When you have multiple workspaces, you can create a workspace settings called global to apply settings across all workspaces. Keep in mind that you still need to create a separate workspace settings per workspace to configure additional workspace-specific details, such as import and export rules.

The following example enables service isolation so that services are secured by default. However, federation is disabled because it can impact performance if every service in the workspace is federated across namespaces.
```
kubectl apply --context $MGMT_CONTEXT -n gloo-mesh -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
 name: global
 namespace: gloo-mesh
spec:
 options:
   serviceIsolation:
     enabled: true
   federation:
     enabled: false
EOF
```
Give the app owner for the team the workspace details, or viewer permission for the resource in the management cluster.

Step 3: Configure workspace settings

As an app owner for your team, you configure the workspace that your platform administrator set up for you by creating a WorkspaceSettings resource. For more information, see Workspace settings configuration.

Each workspace must have only one of its own workspace settings resource. You still need one workspace settings per workspace even if your platform admin set up global settings.

Get access from your platform admin to the namespace and cluster in the workspace where you want to create the workspace settings.
- In most cases, you use a namespace with your team name, such as ops-team in the management cluster.
  Click to expand information for more complex setups.
  Some teams might have a more complex setup where you create Gloo custom resources in the workload clusters. Check with your platform admin to see which clusters and namespaces are included in your workspace. Or, if you have access to the management cluster, run the following command.
  kubectl get workspaces -n gloo-mesh --context $MGMT_CONTEXT -o=jsonpath='{.items[*].spec.workloadClusters}'
Decide which workspaces you want to export your resources to. Get their labels from the platform admin or the other team's app owners.
Decide which workspaces you want to import resources from. Get their labels from the platform admin or the other team's app owners.
Decide if you want to restrict importing or exporting to certain resources, such as Kubernetes services.
Decide if you want to set up service isolation for your workspace. When service isolation is enabled, only services within the same workspace and services that are imported to the workspace can communicate with each other by default. Communication to services outside the workspace is blocked. However, to build a zero-trust architecture, you might want to disable service isolation and instead set up more fine-grained access control with access policies and trim proxy config policies. For more information, see Service isolation.
Decide if you want to set up federation for multicluster routing across your workspace. At the workspace level, federation creates ungrouped Istio service entries in each cluster so that services are accessible across clusters. For single clusters or to improve performance at scale in large environments, disable federation and use virtual destinations instead. For more information, see Federation.

Create the workspace settings with the information that you just gathered, in the cluster and namespace that you want. The following example creates workspace settings for three workspaces for the Ops, Web, and Backend API teams. For more information, see Example workspace YAML configuration.

Ops team
Web team
Backend APIs


kubectl apply --context $MGMT_CONTEXT -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
  name: ops-team
  namespace: ops-team
spec:
  importFrom:
  - workspaces:
    - name: web-team
  exportTo:
  - workspaces:
    - name: "*"
    resources:
    - kind: SERVICE
      namespace: gloo-mesh-addons
    - kind: VIRTUAL_DESTINATION
      namespace: gloo-mesh-addons
  options:
    federation:
      enabled: false
      serviceSelector:
      - namespace: gloo-mesh-addons
    eastWestGateways:
    - selector:
        labels:
          istio: eastwestgateway
    serviceIsolation:
      enabled: true
      trimProxyConfig: true
EOF


kubectl apply --context $MGMT_CONTEXT -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
  name: web-team
  namespace: web-team
spec:
  importFrom:
  - workspaces:
    - name: backend-apis-team
  - workspaces:
    - name: ops-team
  exportTo:
  - workspaces:
    - name: ops-team
  options:
    eastWestGateways:
    - selector:
        labels:
          istio: eastwestgateway
    federation:
      enabled: false
      serviceSelector:
      - namespace: web-ui
    serviceIsolation:
      enabled: true
      trimProxyConfig: true
EOF


kubectl apply --context $MGMT_CONTEXT -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
  name: backend-apis-team
  namespace: backend-apis-team
spec:
  exportTo:
  - workspaces:
    - name: web-team
  importFrom:
  - workspaces:
    - name: ops-team
  options:
    eastWestGateways:
    - selector:
        labels:
          istio: eastwestgateway
    federation:
      enabled: false
      serviceSelector:
      - namespace: backend-apis
    serviceIsolation:
      enabled: true
      trimProxyConfig: true
EOF

Good job! Now that you created your workspace and workspace settings, verify your workspace setup.

Verify your workspace setup

To verify your workspace setup, you can try to import and export Gloo resources. To do so, you need access to at least two different workspaces that are set up to import and export to each other. You might have this access yourself, or work with another team's app owner to verify.

In your workspace settings, make sure that you export to and import the other workspace, such as in the following example.

kubectl get workspacesettings default -n ${WS_NAMESPACE} --context ${REMOTE_CONTEXT} -o=jsonpath='{.spec}'

Example output:

{"exportTo":[{"selector":{"team":"backend"}}, "imports":[{"selector":{"team":"frontend"}}}

Repeat the previous step for the other workspace. Make sure to use the right ${WS_NAMESPACE} and ${REMOTE_CONTEXT} values for that workspace.

Create a Gloo resource to export. Make sure that your workspace settings export the resource, and that the workspace settings of the other workspace import the resource, such as via a label. The following example is for a virtual destination.

kubectl apply --context ${REMOTE_CONTEXT} -n ${WS_NAMESPACE} -f - << EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualDestination
metadata:
  name: istio-ingressgateway
  namespace: ${WS_NAMESPACE}
spec:
  hosts:
  - reviews.global
  ports:
  - name: http
    number: 9080
    protocol: HTTP
  services:
  - labels:
      app: reviews
EOF

Open the Gloo UI.
- meshctl: For more information, see the CLI documentation.
```
meshctl dashboard --kubecontext ${MGMT_CONTEXT}
```
- kubectl:
  1. Port-forward the gloo-mesh-ui service on 8090.
```
kubectl port-forward -n gloo-mesh svc/gloo-mesh-ui 8090:8090 --context ${MGMT_CONTEXT}
```
  2. Open your browser and connect to http://localhost:8090.
Check your workspace import and export settings.
1. From the Overview tab, expand your workspaces to see an overview of the import and export settings, such as in the following figure.
  
  Figure of workspaces overview in the Gloo UI. Note the **Imports / Exports** section.
2. Click MORE DETAILS and review the Workspace Dependencies and the tabbed section of Destination, Routing, and Policies.
  
  Figure of workspaces detail in the Gloo UI. Note that you can review available resources to import.

What's next?

Now that you set up your workspaces, your teams can deploy their apps. Tell them the clusters and namespaces that are part of their workspaces. They can create their Kubernetes deployments and services. Istio automatically adds the services to the mesh.

As part of your team setup, you might want to set up role-based access control (RBAC) for your Kubernetes and Gloo resources.

Then, you can create Gloo resources to control incoming traffic for your workloads. For example, you might set up the following resources.

Request routing with route tables.
Policies, such as retries or timeouts, that can be reused across routes and destinations.