Installing Gloo Edge to Multiple Namespaces

In the default deployment scenario, a single deployment of the Gloo Edge control plane and Envoy proxy are installed for the entire cluster. However, in some cases, it may be desirable to deploy multiple instances of the Gloo Edge control plane and proxies in a single cluster.

This is useful when multiple tenants or applications want control over their own instance of Gloo Edge. Some deployment scenarios may involve a Gloo Edge per-application architecture. Additionally, different Gloo Edge instances living in their own namespace may be given different levels of RBAC permissions.

In this document, we will review how to deploy multiple instances of Gloo Edge to their own namespaces within a single Kubernetes cluster.

Scoping Gloo Edge to specific namespaces

When using the default installation, Gloo Edge will watch all namespaces for Kubernetes services and Gloo Edge CRDs. This means that any Kubernetes service can be a destination for any VirtualService in the cluster.

Gloo Edge can be configured to only watch specific namespaces, meaning Gloo Edge will not see services and CRDs in any namespaces other than those provided in the watchNamespaces setting .

By leveraging this option, we can install Gloo Edge to as many namespaces we need, ensuring that the watchNamespaces do not overlap.

Currently, installing Gloo Edge with specific watchNamespaces requires installation via the Helm chart.

Installing Namespace-Scoped Gloo Edge with Helm

In this section we’ll deploy Gloo Edge twice, each instance to a different namespace, with two different Helm value files. For Gloo Edge Enterprise users, we have included a settings for the Grafana RBAC configuration. Gloo Edge Open Source users can safely remove those settings.

Create a file named gloo1-overrides.yaml and paste the following inside:

settings:
  create: true
  writeNamespace: gloo1
  watchNamespaces:
  - default
  - gloo1
grafana: # The grafana settings can be removed for Gloo Edge OSS
  rbac:
    namespaced: true

Now, let’s install Gloo Edge. Review our Kubernetes installation guide if you need a refresher.

First create the namespace for our first Gloo Edge deployment:

kubectl create ns gloo1

Then install Gloo Edge using one of the following methods:

glooctl install gateway -n gloo1 --values gloo1-overrides.yaml

helm install gloo gloo/gloo --namespace gloo1 -f gloo1-overrides.yaml

Check that gloo pods are running:

kubectl get pod -n gloo1

NAME                             READY   STATUS    RESTARTS   AGE
discovery-798cdd5499-z7rrt       1/1     Running   0          37s
gateway-5fc999b847-jf4xp         1/1     Running   0          32s
gateway-proxy-67f4c7dfb6-hc5kg   1/1     Running   0          27s
gloo-dd5bcdc8f-bvtjh             1/1     Running   0          39s

And we should see that Gloo Edge is only creating Upstreams from services in default and gloo1:

kubectl get us -n gloo1                                              

NAME                      AGE
default-kubernetes-443    1h
gloo1-gateway-proxy-443   1h
gloo1-gateway-proxy-80    1h
gloo1-gloo-9977           1h

Let’s repeat the above process, substituting gloo2 for gloo1:

Create a file named gloo2-overrides.yaml and paste the following inside:

settings:
  create: true
  writeNamespace: gloo2
  watchNamespaces:
  - default
  - gloo2
grafana: # The grafana settings can be removed for Gloo Edge OSS
  rbac:
    namespaced: true

Now, let’s install Gloo Edge for the second time. First create the second namespace:

# create the namespace for our second gloo deployment
kubectl create ns gloo2

Then perform the second installation using one of the following methods:

glooctl install gateway -n gloo2 --values gloo2-overrides.yaml

helm install gloo gloo/gloo --namespace gloo2 -f gloo2-overrides.yaml

Check that gloo pods are running:

kubectl get pod -n gloo2

NAME                             READY   STATUS    RESTARTS   AGE
discovery-798cdd5499-kzmkc       1/1     Running   0          8s
gateway-5fc999b847-pn2tk         1/1     Running   0          8s
gateway-proxy-67f4c7dfb6-284wv   1/1     Running   0          8s
gloo-dd5bcdc8f-krp5p             1/1     Running   0          9s

And we should see that the second installation of Gloo Edge is only creating Upstreams from services in default and gloo2:

kubectl get us -n gloo2

NAME                      AGE
default-kubernetes-443    53s
gloo2-gateway-proxy-443   53s
gloo2-gateway-proxy-80    53s
gloo2-gloo-9977           53s

And that’s it! We can now create routes for Gloo Edge #1 by creating our Virtual Services in the gloo1 namespace, and routes for Gloo Edge #2 by creating Virtual Services in the gloo2 namespace. We can add watchNamespaces to our liking; the only catch is that a Virtual Service which lives in a shared namespace will be applied to both gateways (which can lead to undesired behavior if this was not the intended effect).