Custom Resource Usage

Gloo Edge uses native constructs in Kubernetes to store data, including ConfigMaps, Secrets, and Custom Resource (CRs). This document is meant to summarize what CRs are created by Gloo Edge during installation, and how these CRs interact with the other Gloo Edge objects to store configuration data in the cluster.


Overview

When Gloo Edge is installed on Kubernetes, it creates a number of Custom Resource Definitions that Gloo Edge can use to store data. The following table describes each Custom Resource Definition, its grouping, and its purpose.

Name Grouping Purpose
Settings gloo.solo.io Global settings for all Gloo Edge containers.
Gateway gateway.solo.io Describes a single Listener and the routing Upstreams reachable via the Gateway Proxy.
VirtualService gateway.solo.io Describes the set of routes to match for a set of domains with a destination of a Route Table, Upstream, or Upstream Group.
RouteTable gateway.solo.io Child Routing object for the Gloo Edge gateway.
Proxy gloo.solo.io A combination of Gateway resources to be parsed by Gloo Edge pods.
Upstream gloo.solo.io Upstreams represent destinations for routing requests.
UpstreamGroup gloo.solo.io Group multiple Upstreams and/or external endpoints to be referenced by Virtual Service(s).
AuthConfig enterprise.gloo.solo.io User-facing authentication configuration referenced by Virtual Service(s).

As a quick refresher, Gloo Edge is deployed as pods from four different container images:

The gateway, gloo, and discovery pods act as the control plane for Gloo Edge. The data plane is handled by the gateway-proxy/ingress-proxy pods running Envoy.

Gateway is responsible for:

Gloo Edge is responsible for:

Discovery is responsible for:

The next few sections detail different scenarios where a Custom Resource is used.


Gateway and Proxy Configuration

Virtual Services, Route Tables and Gateway information are all merged together to form a Proxy configuration that the Gloo Edge pods can use to prepare a snapshot for the Envoy Proxy clusters using the translation engine and xDS server on the Gloo Edge pods.

Gateway and Proxy Configuration

When a user or process wants to perform CRUD (Create, Read, Update, Delete) operations on a Virtual Server, Gateway, or Route Table they may use the glooctl command-line tool or kubectl directly to make changes. The changes are written to a new or existing Custom Resource matching the resource type that is being altered. The Gateway pods of Gloo Edge take the information from all three Custom Resource types, and merge and transform the data to create a Proxy Custom Resource. The Proxy Custom Resource is watched by the Gloo Edge pods, which use it to generate the snapshot to be pulled by the Envoy Proxy instances.


Upstreams and Upstream Groups

Upstreams are destinations for traffic sent to the Gloo Edge gateway. A Virtual Service or Route Table may reference one of more Upstreams as destinations. Multiple Upstreams can be combined into an Upstream Group with a list of Upstreams and weights for each Upstream.

Upstreams

Upstreams can be added manually by a user or process, or they can be added automatically through Service Discovery. In the case of a manual addition, a user or process utilizes the glooctl command-line tool or kubectl directly to perform CRUD operations on an Upstream. The Gloo Edge pod is constantly watching the Upstream Custom Resources to see if a change has been made.

Gateway and Proxy Configuration

In the case of automatic addition through discovery, the user or process will deploy a new service to the Kubernetes cluster. The Discovery component will watch for new services being introduced using the Kubernetes integration. When the new service is discovered, the Discovery pod will create a new Custom Resource including details about the new service. The Gloo Edge pod is constantly watching the Upstream Custom Resources to see if a change has been made.

Gateway and Proxy Configuration

Upstream Groups

Upstream Groups are an abstraction used to group multiple Upstreams together and include weights for load-balancing across the Upstreams.

Gateway and Proxy Configuration

The Upstream Group Custom Resource is created by a user or process utilizing the glooctl command-line tool or kubectl directly. The Upstream Group will reference existing Upstream Custom Resources that have already been configured. The Gloo Edge pod is constantly watching the Upstream Group Custom Resources to see if a change has been made.


Settings

Gloo Edge keeps global settings stored in a Settings Custom Resource. When a new Gateway, Gloo Edge, or Discovery pod is created, it will look for a Settings Custom Resource to load its configuration.

Gateway and Proxy Configuration

The Settings Custom Resource is typically created through an installation process using Helm. The values in the CR can be manipulated using the glooctl command-line tool or kubectl directly. The pods run a periodic sync process that looks for changes to the Settings CR. When a change is detected it is applied after an internal snapshot is taken.


Next Steps