Set up multitenancy with workspaces
Gloo introduces a new concept for Kubernetes-based multitenancy, the Workspace custom resource. A workspace consists of one or more Kubernetes namespaces that are in one or more clusters. Think of a workspace as the boundary of your team's resources. To get started, you can create a workspace for each of your teams. Your teams might start with their apps in a couple Kubernetes namespaces in a single cluster. As your teams scale across namespaces and clusters, their workspaces scale with them. For more information, see the Workspaces concept.
Before you begin
-
Install Gloo in your management cluster and register workload clusters.
When you register workload clusters, make sure to use labels in themetadata.labels
section of theKubernetesCluster
resource. Your workspace uses these labels to decide which clusters are part of the workspace. Gloo even detects when you register more clusters with the same label down the road, and adds them to the workspace! -
Set the names of your clusters from your infrastructure provider.
export MGMT_CLUSTER=<management_cluster_name> export REMOTE_CLUSTER=<remote_cluster_name>
-
Save the kubeconfig contexts for your clusters. Run
kubectl config get-contexts
, look for your cluster in theCLUSTER
column, and get the context name in theNAME
column. Note: Do not use context names with underscores. The context name is used as a SAN specification in the generated certificate that connects workload clusters to the management cluster, and underscores in SAN are not FQDN compliant. You can rename a context by runningkubectl config rename-context "<oldcontext>" <newcontext>
.export MGMT_CONTEXT=<management-cluster-context> export REMOTE_CONTEXT=<remote-cluster-context>
-
Review existing workspaces to note which namespaces are already part of a workspace. Namespaces can belong to only one workspace.
kubectl get workspaces -n gloo-mesh --context $MGMT_CONTEXT -o=jsonpath="{.items[*]['metadata.name', 'spec.workloadClusters']}"
The following example shows one
gloo
workspace that selects all clusters ("name":"*"
) and namespaces ("namespaces":[{"name":"*"}]
). If you have a similar workspace, you must modify the workspace after adding new workspaces so that they do not select the same namespaces.gloo [{"name":"*","namespaces":[{"name":"*"}]}]
-
If you are new to workspaces, review the following resources to understand more about the concepts.
- Workspaces concept, including how workspaces function as resource boundaries and a person-driven setup flow.
- Multicluster federation and isolation guide, which sets up a separate workspace for your Bookinfo app and Istio resources like the gateways.
- API reference
Create a workspace
As a platform administrator, you can create a workspace for each of your teams.
-
Pick a name for the workspace. This name might be the same as your team's name, such as
web-team
, and must be unique. -
In the management cluster, create a namespace for each workspace. Later, each team can create all their Gloo custom resources in this namespace. This way, each team's resources are kept in their own workspaces, but all of the Gloo configuration is kept centrally located in the management cluster. For example, you might create the following namespaces for three teams:
ops-team
,web-team
, andbackend-apis-team
.kubectl create namespace --context $MGMT_CONTEXT ops-team kubectl create namespace --context $MGMT_CONTEXT web-team kubectl create namespace --context $MGMT_CONTEXT backend-apis-team
-
Come up with a label for your workspace. Later, other workspaces can use this label to import or export to your workspace. You might want to set up a naming convention for all of your workspaces. For example, you might use
gloo.solo.io/global
for workspaces that are available to any other workspace. -
Decide which clusters you want the workspace to include. A workspace can have many clusters. Also, the same cluster might belong to several workspaces. Instead of selecting individual clusters by name, use labels. Labels allow you to treat clusters “like cattle.” Gloo adds or removes clusters with the same label to your workspace. Workspace labels must match the
KubernetesCluster
labels on the management cluster. You can check those labels with the following command.kubectl get kubernetescluster -n gloo-mesh --context $MGMT_CONTEXT -o=jsonpath="{.items[*]['metadata.name', 'metadata.labels']}"
-
Decide which Kubernetes namespaces across clusters you want the workspace to include. Unlike a cluster, a namespace can belong to only one workspace. To list existing namespaces, you can use the following command.
kubectl get ns --context $MGMT_CONTEXT && kubectl get ns --context $REMOTE_CONTEXT
-
Create the workspace with the information that you just gathered. Note that you must create the workspace resource in the
gloo-mesh
namespace of the management cluster. For an example of three workspaces for the Ops, Web, and Backend API teams, see Example workspace YAML configuration.kubectl apply --context $MGMT_CONTEXT -f- <<EOF apiVersion: admin.gloo.solo.io/v2 kind: Workspace metadata: name: ops-team namespace: gloo-mesh labels: gloo.solo.io/team: 'ops' spec: workloadClusters: - name: 'mgmt' namespaces: - name: ops-team - name: '*' namespaces: - name: istio-gateways - name: gloo-mesh-addons EOF
kubectl apply --context $MGMT_CONTEXT -f- <<EOF apiVersion: admin.gloo.solo.io/v2 kind: Workspace metadata: name: web-team namespace: gloo-mesh labels: gloo.solo.io/team: 'web' spec: workloadClusters: - name: 'mgmt' namespaces: - name: web-team - name: '*' namespaces: - name: web-ui EOF
kubectl apply --context $MGMT_CONTEXT -f- <<EOF apiVersion: admin.gloo.solo.io/v2 kind: Workspace metadata: name: backend-apis-team namespace: gloo-mesh labels: gloo.solo.io/team: 'backend-apis' spec: workloadClusters: - name: 'mgmt' namespaces: - name: backend-apis-team - name: '*' namespaces: - name: backend-apis EOF
-
Optionally, you can configure global workspace settings. These settings apply by default to each workspace in your Gloo environment. Later, the app owner for each workspace can override the defaults in their own workspace settings. For more information, see Workspace configuration.
You must create the global workspace settings resource in thegloo-mesh
namespace of the management cluster, and name the resourceglobal
. Keep in mind that each workspace still needs its own unique workspace settings resource, which your app owner configures in the next section.The following example enables service isolation so that services are secured by default. However, federation is disabled because it can impact performance if every service in the workspace is federated across namespaces.
kubectl apply --context $MGMT_CONTEXT -n gloo-mesh -f- <<EOF apiVersion: admin.gloo.solo.io/v2 kind: WorkspaceSettings metadata: name: global namespace: gloo-mesh spec: options: serviceIsolation: enabled: true federation: enabled: false EOF
-
Optional: After you set up your workspaces, modify any workspaces that might have conflicting namespaces, which you identified before you began. For example, you might have a workspace that selects all namespaces by default, that you can modify to become a management-only workspace.
kubectl edit workspace gloo --context $MGMT_CONTEXT apiVersion: admin.gloo.solo.io/v2 kind: Workspace metadata: name: gloo namespace: gloo-mesh spec: workloadClusters: - name: '$MGMT_CLUSTER' namespaces: - name: 'gloo-mesh'
-
Give the app owner for the team the workspace details, or viewer permission for the resource in the management cluster.
Configure workspace settings
As an app owner for your team, you configure the workspace that your platform administrator set up for you by creating a WorkspaceSettings resource. For more information, see Workspace settings configuration.
- Get access from your platform admin to the namespace and cluster in the workspace where you want to create the workspace settings.
- In most cases, you use a namespace with your team name, such as
ops-team
in the management cluster.
- In most cases, you use a namespace with your team name, such as
- Decide which workspaces you want to export your resources to. Get their labels from the platform admin or the other team's app owners.
- Decide which workspaces you want to import resources from. Get their labels from the platform admin or the other team's app owners.
- Decide if you want to restrict importing or exporting to certain resources, such as Kubernetes services.
- Decide if you want to set up service isolation for your workspace. Enabling service isolation is generally a good idea so that services within or imported to the workspace can communicate with each other by default. But services outside the workspace are blocked. However, you might disable service isolation if you want to build your own zero-trust architecture. For more information, see Service isolation.
- Decide if you want to set up federation for multicluster routing across your workspace. To improve performance in large environments, disable federation and use virtual destinations instead. For more information, see Federation.
- Create the workspace settings with the information that you just gathered, in the cluster and namespace that you want. The following example creates workspace settings for three workspaces for the Ops, Web, and Backend API teams. For more information, see Example workspace YAML configuration.
kubectl apply --context $MGMT_CONTEXT -f- <<EOF apiVersion: admin.gloo.solo.io/v2 kind: WorkspaceSettings metadata: name: ops-team namespace: ops-team spec: importFrom: - workspaces: - name: web-team exportTo: - workspaces: - name: "*" resources: - kind: SERVICE namespace: gloo-mesh-addons - kind: VIRTUAL_DESTINATION namespace: gloo-mesh-addons options: federation: enabled: false serviceSelector: - namespace: gloo-mesh-addons eastWestGateways: - selector: labels: istio: eastwestgateway serviceIsolation: enabled: true trimProxyConfig: true EOF
kubectl apply --context $MGMT_CONTEXT -f- <<EOF apiVersion: admin.gloo.solo.io/v2 kind: WorkspaceSettings metadata: name: web-team namespace: web-team spec: importFrom: - workspaces: - name: backend-apis-team - workspaces: - name: ops-team exportTo: - workspaces: - name: ops-team options: eastWestGateways: - selector: labels: istio: eastwestgateway federation: enabled: false serviceSelector: - namespace: web-ui serviceIsolation: enabled: true trimProxyConfig: true EOF
kubectl apply --context $MGMT_CONTEXT -f- <<EOF apiVersion: admin.gloo.solo.io/v2 kind: WorkspaceSettings metadata: name: backend-apis-team namespace: backend-apis-team spec: exportTo: - workspaces: - name: web-team importFrom: - workspaces: - name: ops-team options: eastWestGateways: - selector: labels: istio: eastwestgateway federation: enabled: false serviceSelector: - namespace: backend-apis serviceIsolation: enabled: true trimProxyConfig: true EOF
Verify your workspace setup
To verify your workspace setup, you can try to import and export Gloo resources. To do so, you need access to at least two different workspaces that are set up to import and export to each other. You might have this access yourself, or work with another team's app owner to verify.
-
In your workspace settings, make sure that you export to and import the other workspace, such as in the following example.
kubectl get workspacesettings default -n ${WS_NAMESPACE} --context ${REMOTE_CONTEXT} -o=jsonpath='{.spec}'
Example output:
{"exportTo":[{"selector":{"team":"backend"}}, "imports":[{"selector":{"team":"frontend"}}}
-
Repeat the previous step for the other workspace. Make sure to use the right
${WS_NAMESPACE}
and${REMOTE_CONTEXT}
values for that workspace. -
Create a Gloo resource to export. Make sure that your workspace settings export the resource, and that the workspace settings of the other workspace import the resource, such as via a label. The following example is for a virtual destination.
kubectl apply --context ${REMOTE_CONTEXT} -n ${WS_NAMESPACE} -f - << EOF apiVersion: networking.gloo.solo.io/v2 kind: VirtualDestination metadata: name: test-details namespace: ${WS_NAMESPACE} labels: export: true spec: hosts: - reviews.global ports: - name: http number: 9080 protocol: HTTP services: - labels: app: reviews EOF
-
Open the Gloo UI.
meshctl
: For more information, see the CLI documentation.meshctl dashboard --kubecontext ${MGMT_CONTEXT}
kubectl
:- Port-forward the
gloo-mesh-ui
service on 8090.kubectl port-forward -n gloo-mesh svc/gloo-mesh-ui 8090:8090 --context ${MGMT_CONTEXT}
- Open your browser and connect to http://localhost:8090.
- Port-forward the
-
Check your workspace import and export settings.
- From the Overview tab, expand your workspaces to see an overview of the import and export settings, such as in the following figure.
Figure of workspaces overview in the Gloo UI. Note the **Imports / Exports** section. - Click MORE DETAILS and review the Workspace Dependencies and the tabbed section of Destination, Routing, and Policies.
Figure of workspaces detail in the Gloo UI. Note that you can review available resources to import.
- From the Overview tab, expand your workspaces to see an overview of the import and export settings, such as in the following figure.
What's next?
Now that you set up your workspaces, your teams can deploy their apps. Tell them the clusters and namespaces that are part of their workspaces. They can create their Kubernetes deployments and services. Istio automatically adds the services to the mesh.
As part of your team setup, you might want to set up role-based access control (RBAC) for your Kubernetes and Gloo resources.
Then, you can create Gloo resources to manage traffic to the services. For example, you might set up the following resources.
- Request routing with route tables and virtual destinations.
- Policies, such as retries or timeouts, that can be reused across routes and destinations.