Custom Resource Usage

Gloo Gateway uses native constructs in Kubernetes to store data, including ConfigMaps, Secrets, and Custom Resources (CRs). This document is meant to summarize what CRs are created by Gloo Gateway during installation, and how these CRs interact with the other Gloo Gateway objects to store configuration data in the cluster.


Overview

When Gloo Gateway is installed on Kubernetes, it creates a number of Custom Resource Definitions that Gloo Gateway can use to store data. The following table describes each Custom Resource Definition, its grouping, and its purpose.

Name Grouping Purpose
Settings gloo.solo.io Global settings for all Gloo Gateway containers.
Gateway gateway.solo.io Describes a single Listener and the routing Upstreams reachable via the Gateway Proxy.
VirtualService gateway.solo.io Describes the set of routes to match for a set of domains with a destination of a Route Table, Upstream, or Upstream Group.
RouteTable gateway.solo.io Child Routing object for Gloo Gateway’s Gateway custom resource.
Proxy gloo.solo.io A combination of Gateway resources to be parsed by Gloo Gateway pods.
Upstream gloo.solo.io Upstreams represent destinations for routing requests.
UpstreamGroup gloo.solo.io Group multiple Upstreams and/or external endpoints to be referenced by Virtual Service(s).
AuthConfig enterprise.gloo.solo.io User-facing authentication configuration referenced by Virtual Service(s).

As a quick refresher, Gloo Gateway is deployed as pods from three different container images:

The gloo and discovery pods act as the control plane for Gloo Gateway. The data plane is handled by the gateway-proxy/ingress-proxy pods running Envoy.

The gloo deployment is responsible for:

Discovery is responsible for:

The next few sections detail different scenarios where a Custom Resource is used.


Gateway and Proxy Configuration

Virtual Services, Route Tables and Gateway information are all merged together to form a Proxy configuration that the Gloo Gateway pods can use to prepare a snapshot for the Envoy Proxy clusters using the translation engine and xDS server on the Gloo Gateway pods.

Gateway and Proxy Configuration

When a user or process wants to perform CRUD (Create, Read, Update, Delete) operations on a Virtual Server, Gateway, or Route Table they may use the glooctl command-line tool or kubectl directly to make changes. The changes are written to a new or existing Custom Resource matching the resource type that is being altered. The Gateway functionality in the Gloo Gateway pods takes the information from all three Custom Resource types, and merges and transforms the data to create a Proxy Custom Resource. The Proxy Custom Resource is watched by the Gloo Gateway pods, which use it to generate the snapshot to be pulled by the Envoy Proxy instances.


Upstreams and Upstream Groups

Upstreams are destinations for traffic sent to the gateway. A Virtual Service or Route Table may reference one of more Upstreams as destinations. Multiple Upstreams can be combined into an Upstream Group with a list of Upstreams and weights for each Upstream.

Upstreams

Upstreams can be added manually by a user or process, or they can be added automatically through Service Discovery. In the case of a manual addition, a user or process utilizes the glooctl command-line tool or kubectl directly to perform CRUD operations on an Upstream. The Gloo Gateway pod is constantly watching the Upstream Custom Resources to see if a change has been made.

Gateway and Proxy Configuration

In the case of automatic addition through discovery, the user or process will deploy a new service to the Kubernetes cluster. The Discovery component will watch for new services being introduced using the Kubernetes integration. When the new service is discovered, the Discovery pod will create a new Custom Resource including details about the new service. The Gloo Gateway pod is constantly watching the Upstream Custom Resources to see if a change has been made.

Gateway and Proxy Configuration

Upstream Groups

Upstream Groups are an abstraction used to group multiple Upstreams together and include weights for load-balancing across the Upstreams.

Gateway and Proxy Configuration

The Upstream Group Custom Resource is created by a user or process utilizing the glooctl command-line tool or kubectl directly. The Upstream Group will reference existing Upstream Custom Resources that have already been configured. The Gloo Gateway pod is constantly watching the Upstream Group Custom Resources to see if a change has been made.


Settings

Gloo Gateway keeps global settings stored in a Settings Custom Resource. When a new Gloo Gateway or Discovery pod is created, it looks for a Settings Custom Resource to load its configuration.

The Settings Custom Resource is typically created through an installation process using Helm. The values in the CR can be manipulated using the glooctl command-line tool or kubectl directly. The pods run a periodic sync process that looks for changes to the Settings CR. When a change is detected it is applied after an internal snapshot is taken.


Next Steps