Prepare to upgrade

Before you upgrade Gloo Gateway, complete the following preparatory steps:

• Prepare your environment, such as upgrading your current version to the latest patch and upgrading any dependencies to the required supported versions.
• Review important changes made to Gloo Gateway in version 1.20, including CRD, Helm, CLI, and feature changes.
• Review frequently-asked questions about the upgrade process.

Prepare your environment

Review the following preparatory steps that might be required for your environment.

Upgrade your current minor version to the latest patch

Before you upgrade your minor version, first upgrade your current version to the latest patch. For example, if you currently run Gloo Gateway Enterprise version 1.19.8, first upgrade your installation to version 1.19.9. This ensures that your current environment is up-to-date with any bug fixes or security patches before you begin the minor version upgrade process.

1. Find the latest patch of your minor version by checking the Open Source changelog or Enterprise changelog.
2. Go to the documentation set for your current minor version. For example, if you currently run Gloo Gateway Enterprise version 1.19.8, use the drop-down menu in the header of this page to select v1.19.x.
3. Follow the upgrade guide, using the latest patch for your minor version.

If required, perform incremental minor version updates

If you plan to upgrade to a version that is more than one minor version greater than your current version, such as to version 1.20 from 1.18 or older, you must upgrade incrementally. For example, you must first use the upgrade guide in the v1.19.x documentation set to upgrade from 1.18 to 1.19, and then follow the upgrade guide in the v1.20.x documentation set to upgrade from 1.19 to 1.20.

Upgrade dependencies

Check that your underlying infrastructure platform, such as Kubernetes, and other dependencies run a version that is supported for 1.20.

1. Review the supported versions for dependencies such as Kubernetes, Istio, Helm, and more.
2. Compare the supported versions against the versions you currently use.
3. If necessary, upgrade your dependencies, such as consulting your cluster infrastructure provider to upgrade the version of Kubernetes that your cluster runs.

Consider settings to avoid downtime

You might deploy Gloo Gateway in Kubernetes environments that use the Kubernetes load balancer, or in non-Kubernetes environments. Depending on your setup, you can take additional steps to avoid downtime during the upgrade process.

• Kubernetes: Enable Envoy readiness and liveness probes during the upgrade. When these probes are set, Kubernetes sends requests only to the healthy Envoy proxy during the upgrade process, which helps to prevent potential downtime. The probes are not enabled in default installations because they can lead to timeouts or other poor getting started experiences.
• Non-Kubernetes: Configure health checks on Envoy. Then, configure your load balancer to leverage these health checks, so that requests stop going to Envoy when it begins draining connections.

Review version 1.20 changes

Review the following changes made to Gloo Gateway in version 1.20. For some changes, you might be required to complete additional steps during the upgrade process.

Breaking changes

Review the breaking changes in this release.

AuthPlugin removed

The AuthPlugin auth config type is removed in 1.20. If you use this config type, you must remove it from your Helm values file before upgrading. If you need to configure your own auth service, check out Custom Auth server.

Discovery disabled by default

To improve performance, discovery is now disabled by default in 1.20. Upstreams are no longer automatically created for discovered Kubernetes services. If you want to continue using discovery, set discovery.enabled=true (or gloo.discovery.enabled in Gloo Gateway Enterprise) in your Helm values file before upgrading. For more information, see the Discovery guide.

Envoy version upgrade

The Envoy dependency in Gloo Gateway 1.20 was upgraded from 1.33.x to 1.35.x. This change includes the following upstream breaking changes. For more information about these changes, see the changelog documentation for Envoy v1.34 and Envoy v1.35.

Envoy v1.34:

• Extproc - Tracing changes: When the modeOverride fields of the header and trailer modes have the value DEFAULT or are unset, no change is made to the processing mode that is set in the filter configuration. Additionally, a bug was fixed in which local replies were incorrectly sent to the extProc server for external processing. You can temoporarily revert this change by setting the runtime guard field envoy.reloadable_features.skip_ext_proc_on_local_reply to false.
• TLS - Envoy FIPS: The Envoy FIPS build is updated to use the same version of BoringSSL as the Envoy standard build to meet the requirements of the revised FedRAMP policy. Default values might have changed in the encyrption settings due to the large version bump.
• HTTP connection manager - Generate request ID: When the x-request-id header is empty, generateRequestId now generates a request ID on the request. Previously, it only generated an ID when the header was missing.
• Formatters: The %CEL% and %METADATA% formatters are now considered built-in, and can be used directly in substitution format strings if the related extensions are linked.

Envoy v1.35:

• Tracing changes: Added max_cache_size to the OpenTelemetry tracer config. This limits the number of spans that can be cached before the cache is flushed. The default is 1024 spans. Previously, flushing only happened at the interval that you set. You can change this setting based on the expected telemetry volume in your environment.

Caching filter deprecated

The caching filter is deprecated and planned to be removed in Gloo Gateway version 1.21.

New features

Change proxy metrics to usedonly stats

By default, the Gloo Gateway Prometheus endpoint emits large numbers of metrics that can overwhelm your Prometheus instance or other instances that scrape these metrics. You can apply a query parameter to the Prometheus scraping endpoint to reduce the number of metrics that the proxy emits.

For more information, see Apply metrics filter to Prometheus scraping endpoint.

Retries on rate limited requests

You can configure a separate retry backoff strategy for requests that are rate limited. This way, you can prevent retries on already rate limited services.

To determine if a request was rate limited, a specific reset header is used, such as the X-RateLimit-Reset or Retry-After headers. If the header is present in the response, the backoff strategy for rate limited requests is applied and the general retry backoff strategy is ignored.

For more information, see Retries for rate limited requests.

Circuit breakers for DFP-enabled routes

You can configure separate circuit breakers for dynamically discovered upstream hosts. By default, Envoy creates a cluster for each resolved upstream and limits the number of connections to this cluster to 1024. When using Dynamic Forward Proxies, Envoy creates a cluster for each host and applies the default circuit breaker settings to it.

Depending on your setup, you might quickly reach the circuit breaker limit for each upstream host, even though overall traffic is not high. To overwrite the default circuit breaker settings, configure the dynamicForwardProxy.circuitBreakers fields on your gateway proxy.

For more information, see Set circuit breakers for dynamically discovered upstreams

Mutual TLS in passthrough auth

You can configure the Gloo Gateway external auth client to use mutual TLS when connecting to the passthrough server. For information for how to set up passthrough auth, see Passthrough auth.

Changelogs

Check the changelogs for the type of Gloo Gateway deployment that you have. Focus especially on any Breaking Changes that might require a different upgrade procedure. For Gloo Gateway Enterprise, you might also review the open source changelogs because most of the proto definitions are open source.

• Open Source changelogs
• Enterprise changelogs: Keep in mind that Gloo Gateway Enterprise pulls in Gloo Gateway Open Source as a dependency. Although the major and minor version numbers are the same for open source and enterprise, their patch versions often differ. For example, open source might use version x.y.a but enterprise uses version x.y.b. If you are unfamiliar with these versioning concepts, see Semantic versioning. Because of the differing patch versions, you might notice different output when checking your version with glooctl version. For example, your API server might run Gloo Gateway Enterprise version 1.20.0, which pulls in Gloo Gateway Open Source version 1.20.0 as a dependency.

  ~ > glooctl version
  Client: {"version":"1.20.0"}
  Server: {"type":"Gateway","enterprise":true,"kubernetes":...,{"Tag":"1.20.0","Name":"grpcserver-ee","Registry":"quay.io/solo-io"},...,{"Tag":"1.20.0","Name":"discovery","Registry":"quay.io/solo-io"},...}
  

Feature changes

GraphQL support removed

In version 1.20.0, support for GraphQL is removed. Any related documentation was also removed. If you need to access GraphQL-specific documentation, such as guides or the API reference, refer to previous documentation versions, such as 1.19.x.

Helm changes

Review the following summary of important new, deprecated, or removed Helm fields. For full details, see the changelogs.

Updated Helm fields:

• discovery.enabled=false: To improve performance, discovery is now disabled by default. This change means that Upstreams are no longer automatically created for discovered Kubernetes services. If you want to continue using discovery, set discovery.enabled=true (or gloo.discovery.enabled in Gloo Gateway Enterprise) in your Helm values file before upgrading. For more information, see the Discovery guide.

Frequently-asked questions

Review the following frequently-asked questions about the upgrade process. If you still aren’t sure about the version upgrade impact, or if your use case doesn’t quite fit the standard upgrade path, feel free to post in the #gloo or #gloo-enterprise channels of our public Slack.

How do I upgrade Gloo Gateway in testing or sandbox environments?

If downtime is not a concern for your use case, you can follow the Quick upgrade guide to update your Gloo Gateway installation.

Note that for sandbox or exploratory environments, the easiest way to upgrade is to uninstall Gloo Gateway by running glooctl uninstall --all. Then, re-install Gloo Gateway at the desired version by the following one of the installation guides.

How do I upgrade Gloo Gateway in a production environment, where downtime is unacceptable?

The basic helm upgrade process is not suitable for environments in which downtime is unacceptable. Instead, you can follow the Canary upgrade guide to deploy multiple version of Gloo Gateway to your cluster, and test the upgrade version before uninstalling the existing version.

Additionally, you might need to take steps to account for other factors such as Gloo Gateway version changes, probe configurations, and external infrastructure like the load balancer that Gloo Gateway uses. Consider setting up liveness probes and healthchecks in your environment.

What happens to my Gloo Gateway CRs during an upgrade? How do I handle breaking changes?

A typical upgrade of Gloo Gateway across minor versions should not cause disruptions to the existing Gloo Gateway state. In the case of a breaking change, Solo will communicate through the upgrade guides, changelogs, or other channels if you must make a specific adjustment to perform the upgrade. Note that you can always use the glooctl debug yaml command to download the current Gloo Gateway state to one large YAML manifest.

Is the upgrade procedure different if I am not a cluster administrator?

If you are not an administrator of your cluster, you might be unable to create custom resource definitions (CRDs) and other cluster-scoped resources, such as cluster roles and cluster role bindings. If you encounter an error related to these resources, you can disable their creation by including the following setting in your Helm values:

global:
  glooRbac:
    create: false

Otherwise, you can try performing an installation of Gloo Gateway that is scoped to a single namespace by including the following setting in your Helm values:

global:
  glooRbac:
    namespaced: true

Why do I get an error about re-creating CRDs when upgrading using helm install or helm upgrade?

Helm v2 does not manage CRDs well, and is not supported in Gloo Gateway. Upgrade to Helm v3, delete the CRDs, and try again.

Why do I get an error about a gateway-certgen job?

The upgrade creates a Kubernetes Job named gateway-certgen to generate a certificate for the validation webhook. The job contains the ttlSecondsAfterFinished value so that the cluster cleans up the job automatically, but because this setting is still in Alpha, your cluster might ignore this value. In this case, you might have an issue while upgrading in which the upgrade attempts to change the gateway-certgen job, but the change fails because the job is immutable. To fix this issue, you can delete the job, which already completed, and re-apply the upgrade.