Distribute gateways (beta)

Distribute your Gateway configuration across multiple clusters.

This feature is an Enterprise-only feature that requires an Enterprise license for either Gloo Gateway or Gloo Mesh.

In Gloo Gateway 1.18 and later, gateway configuration distribution is a beta feature. For more information, see Alpha and beta release process.

About gateway distribution

With Gloo Gateway, you can create API gateways for ingress, egress, waypoint, and other types of traffic management across your clusters. Keeping these gateways in sync across your clusters can be challenging.

Normally, you create many routing resources such as Gateways, HTTPRoutes, and Upstreams in each cluster. You might use a CI/CD integration such as ArgoCD to keep these resources in sync. However, this approach requires an external integration that might not fit into your existing workflows.

By using Gloo Gateway with a multicluster relay setup that is provided by Gloo Mesh (Gloo Platform APIs), you get the ability to distribute gateway configuration across clusters. This way, you can federate your gateway configuration across your clusters without the need for an external integration.

The relay connection consists of a Gloo management server in a source cluster and a Gloo agent in each workload cluster. This way, the management server can send configuration in the source cluster to each of the connected workload clusters. This configuration includes both Kubernetes Gateway API and Gloo Gateway resources, such as Gateways, HTTPRoutes, and Upstreams. The workload clusters have Gloo Gateway installed so that the distributed resources provide the API gateway functionality per cluster.

For example, a distributed Gateway triggers Gloo Gateway to create a gateway proxy for the workload cluster. Then, distributed HTTPRoutes can attach to the distributed Gateway and forward traffic to distributed Upstreams and further to the apps that run in the workload clusters.

How gateway distribution works

Gateway distribution works by using annotations. Click through the following tabs to review diagrams that show how you can use the annotation to distribute resources in your environment.

Config distribution works by using annotations.

In the management cluster, you create a resource, such as a Gateway. You include the special gloo.solo.io/distribute-to annotation on the resource. This annotation instructs the Gloo management server to distribute the Gateway to the selected clusters via the Gloo Mesh (Gloo Platform APIs) relay connection between the management server and the agents in the workload clusters. The wildcard (*) value selects all connected clusters.

In the following example, the Gateway is distributed to both connected clusters in the same namespace as the source. The creation of the Gateway resource triggers the Gloo management server to create a gateway proxy in the same namespace.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gloo.solo.io/distribute-to: "*"
  name: my-gateway
  namespace: default
...

In the gloo.solo.io/distribute-to annotation, you can specify a single cluster, list of clusters (such as "qa, canary"), all clusters with a wildcard (*) value, or no clusters by leaving the value blank (""). Cluster names must match the names in the KubernetesClusters for the connected clusters in the Gloo Mesh (Gloo Platform APIs) Helm installation.

Figure: Overview of namespace distribution

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gloo.solo.io/distribute-to: "cluster1"
  name: my-gateway
  namespace: default
...

The Gateway is distributed as follows:

cluster1: The Gateway is distributed in the same namespace it was created in on the management cluster, the default namespace.
cluster2: The annotation does not include cluster2, so the Gateway is not distributed to the workload cluster.

By default, distribution uses namespace-sameness. This means that resources are distributed from the same namespace that they are created in the management cluster to the workload clusters.

For example, if you create a Gateway in the default namespace in the management cluster, the Gateway is distributed to all workload clusters in the default namespace.

However, you can configure a different namespace in the workload cluster to distribute the resource to. In the gloo.solo.io/distribute-to annotation, include <cluster>/<namespace> in the value, such as the following example. You can also distribute the Gateway to a different namespace in all the clusters by using a wildcard (*) value for the cluster, such as gloo.solo.io/distribute-to: "*/gw".

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gloo.solo.io/distribute-to: "cluster1,cluster2/gw"
  name: my-gateway
  namespace: default
...

The Gateway is distributed as follows:

cluster1: Because no namespace is specified, the Gateway is distributed in the same namespace as it is created in the management cluster, in the default namespace.
cluster2: The annotation specifies a namespace, so the Gateway is distributed to the gw namespace in the workload cluster.

In the previous examples, you distributed a single resource, a Gateway. However, you can distribute multiple and individual resources, such as Gateways, HTTPRoutes, and Upstreams. Apply the gloo.solo.io/distribute-to annotation to each resource that you want to distribute. You can configure the cluster and namespace options for each resource in its annotation. If you don’t want to distribute a resource, omit the annotation or set the value to none.

Note that HTTPRoutes must be backed by an Upstream resource. Other backend kinds, such as Kubernetes Services, are not supported for distribution.

This ability can come in handy when you have cluster or namespace-specific resources to distribute, such as:

Gateways specific to some clusters, such as a handful of egress-only gateways
Local cluster rate limit policies
Routes that are in different namespaces in different clusters due to multitenancy or other reasons

The resources are distributed as follows:

Gateway: The Gateway annotation uses a wildcard (*) value, so the Gateway is distributed to all clusters.
HTTPRoute 1: Route 1 is distributed only to cluster1.
HTTPRoute 2: Route 2 is distributed only to cluster2.
Policy 1: The policy annotation uses a wildcard (*) value, so the policy is distributed to all clusters.
Policy 2: The policy omits the annotation (or sets the value to none), so it is not distributed to any cluster.

Distributable resources

You can distribute the following gateway resources:

Gateways
GatewayClass
GatewayParameters
HTTPRoutes
HTTPRoutes must be backed by an Upstream resource. Other backend kinds, such as Kubernetes Services, are not supported for distribution.
HTTPListenerOptions
ListenerOptions
RouteOptions
VirtualHostOptions
Upstreams

Distributed resource FAQs

Review the following frequently asked questions to guide your usage of the config distribution feature.

What is the name of distributed resources?

When the resource is distributed, the name of the resource is the same as the original resource.

In the workload clusters, the copied resources also get a special annotation, gloo.solo.io/distributed-source. The naming convention for the annotation is as follows:

gloo://<cluster>/ns/<namespace>/kind/<kind>/name/<name>

This way, you can tell which resource controls the distributed copy. You can use this information to help debugging or automate tracing tools.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gloo.solo.io/distributed-source: "gloo://management-cluster/ns/default/kind/Gateway/name/my-gateway"
...

What namespace is the distributed resource created in?

The namespace is the same as the original resource by default, unless otherwise specified in the source annotation.

Can I make changes to my distributed resources?

Yes, but only by changing the source resource in the management cluster.

Any changes made to the resource in the workload cluster are not reflected in the management cluster. Furthermore, the changes are automatically overwritten upon the next relay sync.

What happens if I delete the source resource?

If you delete the source resource, the distributed resource is also deleted.

How should I set up my distributed resources?

Review How it works to plan how to distribute your resources via annotations across clusters and namespaces.

Keep in mind that the config distribution does not check for resource consistency across clusters. For example, you might have a Gateway, HTTPRoute, Upstream, and several policies in the source cluster. If you annotate only the Gateway and policies, the HTTPRoute and Upstream are not distributed to the workload clusters. If you do not have existing HTTPRoutes and Upstreams already in the workload clusters, the routing does not work. Make sure that you distribute all the resources that you need for your routing use case.

Also, be aware of potential conflicts in your workload clusters in more advanced scenarios, such as policy inheritance or route delegation. For example, if you distribute a policy from the management cluster but have a pre-existing policy in the workload cluster, this might cause a conflict.

Before you begin

Create multiple clusters for your distributed gateway scenario.
- Management cluster: One cluster must be the management cluster where you create the resources that you want to distribute to your workload clusters. You install the Gloo management plane into this cluster. You can choose whether to run this cluster as a workload cluster as well, in which case you also install Gloo Gateway so that gateway proxies can be created from your Gateway resources.
- Workload clusters: Other clusters are remote workload clusters that are registered with the management cluster so that they can receive the distributed resources. You run the Gloo Mesh agents and Gloo Gateway in these clusters.

Set environment variables for your management and workload clusters.

export MGMT_CLUSTER=<management-cluster>
export MGMT_CONTEXT=<management-cluster-context>
export REMOTE_CLUSTER=<workload-cluster>
export REMOTE_CONTEXT=<workload-cluster-context>

Set an environment variable for the Gloo Mesh (Gloo Platform APIs) installation. Gateway distribution is supported in version 2.9.6 and later.
```
export GLOO_MESH_VERSION=2.9.6
```
Save either your Gloo Gateway Enterprise or Gloo Mesh (Gloo Platform APIs) license in an environment variable. An Enterprise-level license is required to use the config distribution feature.
- Gloo Gateway:
  export GLOO_GATEWAY_LICENSE_KEY=<gateway-license>
- Gloo Mesh (Gloo Platform APIs):
  export GLOO_MESH_LICENSE_KEY=<mesh-license>
In the workload clusters, create the sample httpbin app.

Step 1: Set up the management cluster

Set up the management cluster with the relay management server and Kubernetes Gateway resources.

Install the Gloo management server in your management cluster with the ConfigDistribution feature gate enabled. The following tabs include a basic example.
Include the following flag in the meshctl install multicluster quickstart.
meshctl install --profiles mgmt-server \ --kubecontext ${MGMT_CONTEXT} \ --set common.cluster=${MGMT_CLUSTER} \ --set licensing.glooMeshLicenseKey=${GLOO_MESH_LICENSE_KEY} \ --set featureGates.ConfigDistribution=true
1. Include the feature gate in your gloo-platform Helm values file. For more information about other options such as certificates to secure the relay connection, see the Gloo Mesh (Gloo Platform APIs) multicluster setup or upgrade docs.
  glooMgmtServer: enabled: true featureGates: ConfigDistribution: true ...
2. Be sure to include the enterprise CRDs in your gloo-platform-crds Helm chart, which are required for config distribution.
  installEnterpriseCrds: true ...
Install the Kubernetes Gateway API CRDs. This way, you can create Kubernetes Gateway resources in your management cluster, including Gateways. Later, these resources are distributed to your remote cluster.
```
kubectl apply --context $MGMT_CONTEXT -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml
```
Choose whether to install Gloo Gateway in the management cluster.
- To run Gloo Gateway workloads in the management cluster, such as creating gateway proxies, install the open source or enterprise edition of Gloo Gateway.
- To distribute Gloo Gateway CRDs such as policies or GatewayParameters, install Gloo Gateway Enterprise. This is required even if you do not plan to run workloads in the management cluster.

Step 2: Set up the workload cluster

Install Gloo Gateway in your workload cluster so that you can automatically create gateway proxies from distributed Gateway resources. Register the workload cluster with the Gloo management server to establish a relay connection with the management cluster.

Install the Kubernetes Gateway API CRDs.

kubectl apply --context $REMOTE_CONTEXT -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml

Install Gloo Gateway Enterprise.
If you have not already done so, install the Gloo agent in your workload cluster and register it with the management cluster.

Verify the relay connection between the management and workload clusters. The management server and agent must be connected so that resources can be successfully distributed from the management cluster to the workload cluster.

meshctl check --kubecontext $MGMT_CONTEXT

Example output:

🟢 Mgmt server connectivity to workload agents

Cluster         | Registered | Connected Pod                                                                    
$REMOTE_CLUSTER | true       | gloo-mesh/gloo-mesh-mgmt-server-76845d886d-fnpsp

Connected Pod                                    | Clusters    
gloo-mesh/gloo-mesh-mgmt-server-76845d886d-fnpsp | 1

Optional: Repeat these steps for each workload cluster that you want to distribute gateway resources to.

Step 3: Distribute gateway configuration

Create the gateway configuration that you want to distribute across clusters in the management cluster. The configuration includes the Gateway itself along with the routing resources that the gateway serves traffic for, such as HTTPRoutes and Upstreams, as shown in the following diagram.

Create a Gateway.

kubectl apply --context $MGMT_CONTEXT -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: distributed-gateway
  namespace: gloo-mesh
  labels:
    example: config-distro
  annotations:
    gloo.solo.io/distribute-to: "*/gloo-system"
spec:
  gatewayClassName: gloo-gateway-distribute
  listeners:
  - protocol: HTTP
    port: 8080
    hostname: mydomain.com
    name: http
    allowedRoutes:
      namespaces:
        from: All
EOF

Setting	Description
`namespace`	The namespace where the Gateway is created. Because the management cluster does not have Gloo Gateway installed, the Gateway is created in the `gloo-mesh` namespace.
`gloo.solo.io/distribute-to: "*/gloo-system"`	The annotation that tells the Gloo management server to copy the resource configuration to the selected clusters and namespaces. The wildcard (`*`) selects all connected clusters for config distribution. You can also specify multiple clusters, such as `"qa, canary"`, or select no clusters by leaving the value blank (`""`). Cluster names must match the name in the KubernetesCluster for the registered cluster. The configuration is distributed to the `gloo-system` namespace in the workload clusters, which differs from the namespace that this Gateway is created in the source cluster.
`gatewayClassName: gloo-gateway-distribute`	The GatewayClass that Gloo uses for distributing configuration. This way, Gloo Gateway does not create a gateway proxy in the management cluster where this Gateway resource is created. Instead, the Gateway configuration is copied to the workload clusters and the GatewayClass field is updated to refer to the `gloo-gateway` GatewayClass. Then, Gloo Gateway automatically creates the gateway proxies in each workload cluster. If you want the gateway proxy to be created in both the management and remote clusters, you can use the regular `gloo-gateway` GatewayClass value.
`listeners`	The listener configuration for the Gateway. The example sets up an HTTP listener on `mydomain.com`, but you can update to use your own.

Create an Upstream to represent the backend service that you want to route traffic to. The following example uses the httpbin app, and as such, selects the httpbin namespace. You don’t need httpbin in your management cluster. Instead, this app runs in the workload cluster that the gateway configuration is distributed to. Note that the Upstream includes the gloo.solo.io/distribute-to annotation, which configures the namespace to the httpbin namespace in the workload cluster where the app runs.
```
kubectl apply --context $MGMT_CONTEXT -f- <<EOF
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: httpbin
  namespace: gloo-mesh
  labels:
    example: config-distro
  annotations:
    gloo.solo.io/distribute-to: "*/httpbin"
spec:
  kube:
    serviceName: httpbin
    serviceNamespace: httpbin
    servicePort: 8000
EOF
```

Create an HTTPRoute that includes the gloo.solo.io/distribute-to annotation. Note that the parentRef in the HTTPRoute refers to the namespace that the Gateway is distributed to, gloo-system, as opposed to the namespace that the source Gateway is created in. This way, when the resources are distributed, the HTTPRoute refers to the correct Gateway.

kubectl apply --context $MGMT_CONTEXT -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin-mydomain
  namespace: gloo-mesh
  labels:
    example: config-distro
  annotations:
    gloo.solo.io/distribute-to: "*/httpbin"
spec:
  parentRefs:
    - name: distributed-gateway
      namespace: gloo-system
  rules:
    - backendRefs:
      - name: httpbin
        kind: Upstream
        group: gloo.solo.io
EOF

Good job! You created gateway configuration that is automatically distributed to the workload clusters.

Step 4: Verify the distributed gateway resources

In your workload cluster, verify that the configuration is distributed. The resources have the same name as the source and include a special annotation.

List the distributed resources in the workload cluster.

kubectl get httproutes,upstreams,gateways --context $REMOTE_CONTEXT -A -l example=config-distro

Example output:

NAME                                                                                 HOSTNAMES   AGE
httproute.gateway.networking.k8s.io/httpbin-mydomain-gateway-distro-distributed-gateway               11d

NAME                                                          AGE
upstream.gloo.solo.io/httpbin-gateway-distro-distributed-gateway   11d

NAME                                                    CLASS          ADDRESS          PROGRAMMED   AGE
gateway.gateway.networking.k8s.io/distributed-gateway   gloo-gateway   34.162.244.194   True         11d

Describe one of the distributed resources to confirm that it follows the naming convention. Names are subject to Kubernetes naming conventions, such as the 63-character limit.

The following naming conventions apply:

Resource name: The same as the source resource name.
gloo.solo.io/distributed-source annotation: A special annotation that includes the source of the distributed resource, in the format gloo://<cluster>/ns/<namespace>/kind/<kind>/name/<name>.

kubectl describe httproute httpbin-mydomain-gateway-distro-distributed-gateway -n gateway-distro --context $REMOTE_CONTEXT

Example output:

Name:         httpbin-mydomain
Namespace:    httpbin
Labels:       agent.gloo.solo.io=gloo-mesh
              cluster.multicluster.solo.io=gloo-gateway-docs-mgt
              example=config-distro
              owner.gloo.solo.io/name=gloo-mesh
              reconciler.mesh.gloo.solo.io/name=gloo-gateway
              relay.solo.io/cluster=gloo-gateway-docs-mgt
Annotations:  gloo.solo.io/distributed-source: gloo://$MGMT_CLUSTER/ns/gloo-mesh/kind/HTTPRoute/name/httpbin-mydomain
API Version:  gateway.networking.k8s.io/v1
Kind:         HTTPRoute
...

Nice! You confirmed that your distributed resources are in the remote cluster.

Step 5: Verify routing behavior

Now that you have distributed gateway configuration, you can verify that the routes work as expected.

Get the external address of the distributed gateway and save it in an environment variable.

export INGRESS_GW_ADDRESS=$(kubectl get svc -n gateway-distro gloo-proxy-distributed-gateway --context $REMOTE_CONTEXT -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS

kubectl port-forward deployment/gloo-proxy-distributed-gateway --context $REMOTE_CONTEXT -n gateway-distro 8080:8080

Send a request to the httpbin app through the distributed gateway.

curl -vik http://$INGRESS_GW_ADDRESS:8080/status/200 -H "host: mydomain.com:8080"

Example output:

HTTP/1.1 200 OK

Next steps

Whenever you want to update the config, you can edit the source config in the management cluster. The change is automatically distributed to the workload clusters. Don’t worry if you make any changes to the distributed config in the workload clusters. These changes are quickly overwritten back so that your distributed config does not drift from the source config.

For ideas of how to set up your gateways, see Deployment patterns.

Cleanup

You can optionally remove the resources that you set up as part of this guide.

Delete the gateway config resources in the management cluster.

kubectl delete httproute,upstream,gateway --context $MGMT_CONTEXT -A -l example=config-distro

Verify that the resources are deleted in the workload clusters.

kubectl get httproute,upstream,gateway --context $REMOTE_CONTEXT -A -l example=config-distro

Optionally disable the config distribution feature gate in your Gloo Mesh (Gloo Platform APIs) installation in the management cluster.

In the mgmt-plane.yaml file, set the featureGates.ConfigDistribution to false.
```
...
featureGates:
  ConfigDistribution: false
```

Upgrade your Helm installation.

helm upgrade -i gloo-platform gloo-platform/gloo-platform \
 --kube-context $MGMT_CONTEXT \
 -n gloo-mesh \
 --version $GLOO_MESH_VERSION \
 --values mgmt-plane.yaml \
 --set common.cluster=$MGMT_CLUSTER \
 --set licensing.glooMeshCoreLicenseKey=$GLOO_MESH_LICENSE_KEY

Optionally uninstall Gloo Mesh (Gloo Platform APIs).

Customize the default setup Deployment patterns