☰
Navigation :

Prepare to upgrade

Before you upgrade Gloo Gateway, complete the following preparatory steps:

Prepare your environment

Review the following preparatory steps that might be required for your environment.

Upgrade your current minor version to the latest patch

Before you upgrade your minor version, first upgrade your current version to the latest patch. For example, if you currently run Gloo Gateway Enterprise version 1.18.10, first upgrade your installation to version 1.18.11. This ensures that your current environment is up-to-date with any bug fixes or security patches before you begin the minor version upgrade process.

  1. Find the latest patch of your minor version by checking the Open Source changelog or Enterprise changelog.
  2. Go to the documentation set for your current minor version. For example, if you currently run Gloo Gateway Enterprise version 1.18.10, use the drop-down menu in the header of this page to select v1.18.x.
  3. Follow the upgrade guide, using the latest patch for your minor version.

If required, perform incremental minor version updates

If you plan to upgrade to a version that is more than one minor version greater than your current version, such as to version 1.19 from 1.17 or older, you must upgrade incrementally. For example, you must first use the upgrade guide in the v1.18.x documentation set to upgrade from 1.17 to 1.18, and then follow the upgrade guide in the v1.19.x documentation set to upgrade from 1.18 to 1.19.

Upgrade dependencies

Check that your underlying infrastructure platform, such as Kubernetes, and other dependencies run a version that is supported for 1.19.

  1. Review the supported versions for dependencies such as Kubernetes, Istio, Helm, and more.
  2. Compare the supported versions against the versions you currently use.
  3. If necessary, upgrade your dependencies, such as consulting your cluster infrastructure provider to upgrade the version of Kubernetes that your cluster runs.

Consider settings to avoid downtime

You might deploy Gloo Gateway in Kubernetes environments that use the Kubernetes load balancer, or in non-Kubernetes environments. Depending on your setup, you can take additional steps to avoid downtime during the upgrade process.

Review version 1.19 changes

Review the following changes made to Gloo Gateway in version 1.19. For some changes, you might be required to complete additional steps during the upgrade process.

Breaking changes

Envoy version upgrade

The Envoy dependency in Gloo Gateway 1.19 was upgraded from 1.31.x to 1.33.x. This change includes the following upstream breaking changes. For more information about these changes, see the Envoy changelog documentation.

New features

Set authority header for gRPC OpenTelemetry collectors

When referencing a gRPC OpenTelemetry collector in your Gateway, Gloo Gateway automatically generates an Envoy configuration that sets the cluster name as the :authority pseudo-header. If your collector expects a different :authority header, you can specify that by setting the spec.httpGateway.options.httpConnectionManagerSettings.tracing.openTelemetryConfig.grpcService.authority value on your Gateway as shown in the following example.

apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  labels:
    app: gloo
    app.kubernetes.io/name: gateway-proxy-tracing-authority
  name: gateway-proxy-tracing-authority
spec:
  bindAddress: '::'
  bindPort: 18082
  proxyNames:
    - gateway-proxy
  httpGateway:
    virtualServiceSelector:
      gateway-type: tracing
    options:
      httpConnectionManagerSettings:
        tracing:
          openTelemetryConfig:
            collectorUpstreamRef:
              name: opentelemetry-collector
              namespace: default
            grpcService:
              authority: my-authority

Log filter state in gRPC access logs

You can enable logging of the filter state when performing gRPC access logging. The filter state logger calls the FilterState::Object::serializeAsProto to serialize the filter state object.

You can enable the filter in your Helm values file or a Gateway resource directly.

The following example adds the modsecurity object from a WAF policy to the filter state object in the access log.

gloo:
  accessLogger:
    enabled: true
    image:
      registry: quay.io/solo-io
      tag: 1.0.0-ci1
  gatewayProxies:
    gatewayProxy:
      gatewaySettings:
        customHttpGateway:
          options:
            waf:
              auditLogging:
                action: ALWAYS
                location: FILTER_STATE
              customInterventionMessage: 'ModSecurity intervention! Custom message details here..'
              ruleSets:
              - ruleStr: |
                  # Turn rule engine on
                  SecRuleEngine On
                  SecRule REQUEST_HEADERS:User-Agent "scammer" "deny,status:403,id:107,phase:1,msg:'blocked scammer'"                  
        accessLoggingService:
          accessLog:
          - grpcService:
              logName: example
              staticClusterName: access_log_cluster
              filterStateObjectsToLog:
              - io.solo.modsecurity.audit_log

The following example adds the modsecurity object from a WAF policy to the filter state object in the access log.

apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  name: gateway-proxy
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8080
  proxyNames: 
    - gateway-proxy
  httpGateway: 
    options:
      waf:
        customInterventionMessage: 'ModSecurity intervention! Custom message details here..'
        ruleSets:
        - ruleStr: |
            # Turn rule engine on
            SecRuleEngine On
            SecRule REQUEST_HEADERS:User-Agent "scammer" "deny,status:403,id:107,phase:1,msg:'blocked scammer'"             
  useProxyProto: false
  options:
    accessLoggingService:
      accessLog:
      - grpcService:
          logName: example
          staticClusterName: access_log_cluster
          filterStateObjectsToLog:
          - io.solo.modsecurity.audit_log

Example access log output:

"filter_state_objects":{"io.solo.modsecurity.audit_log":{"type_url":"type.googleapis.com/google.protobuf.StringValue","value":"Ct0DLS0tQS0tClsxMC9NYXIvMjAyNToxNDoyNTowMSArMDAwMF0gMTc0MTYxNjcwMTQ1LjkwMTMzMCAxMjcuMC4wLjEgMzg5NTQgIDAKLS0tQi0tCkdFVCAvYWxsLXBldHMgSFRUUC8xLjEKOm1ldGhvZDogR0VUCmhvc3Q6IGxvY2FsaG9zdDo4MDgwCjpwYXRoOiAvYWxsLXBldHMKOmF1dGhvcml0eTogbG9jYWxob3N0OjgwODAKeC1mb3J3YXJkZWQtcHJvdG86IGh0dHAKOnNjaGVtZTogaHR0cAp1c2VyLWFnZW50OiBjdXJsLzguMTEuMAphY2NlcHQ6ICovKgp4LXJlcXVlc3QtaWQ6IDUwZTBjZTVmLTk0NjktNGQ3NS1hMzUxLWU4MmI0ODFlYTFmYgoKLS0tRi0tCkhUVFAvMS4xIDIwMAo6c3RhdHVzOiAyMDAKY29udGVudC10eXBlOiBhcHBsaWNhdGlvbi94bWwKZGF0ZTogTW9uLCAxMCBNYXIgMjAyNSAxNDoyNTowMSBHTVQKY29udGVudC1sZW5ndGg6IDg2CngtZW52b3ktdXBzdHJlYW0tc2VydmljZS10aW1lOiAyCgotLS1ILS0KCi0tLVotLQoK"}}}

For more information, see the Access logging API.

Match conditions on validation webhook

You can now specify match conditions on the Gloo Gateway or Kubernetes validating admission webhook level to filter the resources that you want to include or exclude from validation. Match conditions are written in CEL (Common Expression Language).

For example to exclude a secret with the label “foo” from being validated with the Kubernetes webhook, add the following values to your Helm values file.

gateway:
  validation:
    kubeCoreFailurePolicy: Fail # For "strict" validation mode, fail the validation if webhook server is not available
    kubeCoreMatchConditions:
    - name: 'not-a-secret-or-secret-with-foo-label-key'
      expression: '!(request.kind.kind == "Secret" && "labels" in object.metadata && "foo" in object.metadata.labels)'

To exclude these secrets from the Gloo Gateway validation webhook, add the following values.

gateway:
  validation:
    FailurePolicy: Fail # For "strict" validation mode, fail the validation if webhook server is not available
    matchConditions:
    - name: 'not-a-secret-or-secret-with-foo-label-key'
      expression: '!(request.kind.kind == "Secret" && "labels" in object.metadata && "foo" in object.metadata.labels)'

For more information, see Exclude resources from validation.

Send gateway access logs to OTel collector

You can configure your Gateway to send access logs to an OpenTelemetry collector as shown in the following example.

apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  name: gateway-proxy
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8080
  httpGateway: {}
  proxyNames:
  - gateway-proxy
  useProxyProto: false
  options:
    accessLoggingService:
      accessLog:
        - openTelemetryService:
            logName: example
            collector:
              endpoint: otel-collector.default.svc.cluster.local:4317

For more information and additional settings, see the Access log API.

Kubernetes 1.32 support

Starting in version 1.19.0, Gloo Gateway can now run on Kubernetes 1.32. For more information about supported Kubernetes, Envoy, and Istio versions, see Supported versions.

Istio 1.25 support

Starting in version 1.19.0, Gloo Gateway can now run with Istio 1.25. For more information about supported Kubernetes, Envoy, and Istio versions, see Supported versions.

New common category for CRs

A new common category was added to all Gloo Gateway custom resources. This common category allows you to easily list all Gloo Gateway custom resources in your environment.

To list all Gloo Gateway resources across all namespaces:.

kubectl get gloo-gateway -A

To list all enterprise custom resources only:

kubectl get solo-io -A

Changelogs

Check the changelogs for the type of Gloo Gateway deployment that you have. Focus especially on any Breaking Changes that might require a different upgrade procedure. For Gloo Gateway Enterprise, you might also review the open source changelogs because most of the proto definitions are open source.

You can use the changelogs' built-in comparison tool to compare between your current version and the version that you want to upgrade to.

CLI changes

You must upgrade glooctl before you upgrade Gloo Gateway. Because glooctl can create resources in your cluster, such as with glooctl add route, you might have errors in Gloo Gateway if you create resources with an older version of glooctl.

Review the following summary of important new, deprecated, or removed CLI options. For full details, see the changelogs.

New CLI commands or options:

Frequently-asked questions

Review the following frequently-asked questions about the upgrade process. If you still aren’t sure about the version upgrade impact, or if your use case doesn’t quite fit the standard upgrade path, feel free to post in the #gloo or #gloo-enterprise channels of our public Slack.

How do I upgrade Gloo Gateway in testing or sandbox environments?

If downtime is not a concern for your use case, you can follow the Quick upgrade guide to update your Gloo Gateway installation.

Note that for sandbox or exploratory environments, the easiest way to upgrade is to uninstall Gloo Gateway by running glooctl uninstall --all. Then, re-install Gloo Gateway at the desired version by the following one of the installation guides.

How do I upgrade Gloo Gateway in a production environment, where downtime is unacceptable?

The basic helm upgrade process is not suitable for environments in which downtime is unacceptable. Instead, you can follow the Canary upgrade guide to deploy multiple version of Gloo Gateway to your cluster, and test the upgrade version before uninstalling the existing version.

Additionally, you might need to take steps to account for other factors such as Gloo Gateway version changes, probe configurations, and external infrastructure like the load balancer that Gloo Gateway uses. Consider setting up liveness probes and healthchecks in your environment.

What happens to my Gloo Gateway CRs during an upgrade? How do I handle breaking changes?

A typical upgrade of Gloo Gateway across minor versions should not cause disruptions to the existing Gloo Gateway state. In the case of a breaking change, Solo will communicate through the upgrade guides, changelogs, or other channels if you must make a specific adjustment to perform the upgrade. Note that you can always use the glooctl debug yaml command to download the current Gloo Gateway state to one large YAML manifest.

Is the upgrade procedure different if I am not a cluster administrator?

If you are not an administrator of your cluster, you might be unable to create custom resource definitions (CRDs) and other cluster-scoped resources, such as cluster roles and cluster role bindings. If you encounter an error related to these resources, you can disable their creation by including the following setting in your Helm values:

global:
  glooRbac:
    create: false

Otherwise, you can try performing an installation of Gloo Gateway that is scoped to a single namespace by including the following setting in your Helm values:

global:
  glooRbac:
    namespaced: true

Why do I get an error about re-creating CRDs when upgrading using helm install or helm upgrade?

Helm v2 does not manage CRDs well, and is not supported in Gloo Gateway. Upgrade to Helm v3, delete the CRDs, and try again.

Why do I get an error about a gateway-certgen job?

The upgrade creates a Kubernetes Job named gateway-certgen to generate a certificate for the validation webhook. The job contains the ttlSecondsAfterFinished value so that the cluster cleans up the job automatically, but because this setting is still in Alpha, your cluster might ignore this value. In this case, you might have an issue while upgrading in which the upgrade attempts to change the gateway-certgen job, but the change fails because the job is immutable. To fix this issue, you can delete the job, which already completed, and re-apply the upgrade.