Prepare to upgrade
Before you upgrade Gloo Gateway, complete the following preparatory steps:
- Prepare your environment, such as upgrading your current version to the latest patch and upgrading any dependencies to the required supported versions.
- Review important changes made to Gloo Gateway in version 1.19, including CRD, Helm, CLI, and feature changes.
- Review frequently-asked questions about the upgrade process.
Prepare your environment
Review the following preparatory steps that might be required for your environment.
Upgrade your current minor version to the latest patch
Before you upgrade your minor version, first upgrade your current version to the latest patch. For example, if you currently run Gloo Gateway Enterprise version 1.18.10, first upgrade your installation to version 1.18.11. This ensures that your current environment is up-to-date with any bug fixes or security patches before you begin the minor version upgrade process.
- Find the latest patch of your minor version by checking the Open Source changelog or Enterprise changelog.
- Go to the documentation set for your current minor version. For example, if you currently run Gloo Gateway Enterprise version 1.18.10, use the drop-down menu in the header of this page to select v1.18.x.
- Follow the upgrade guide, using the latest patch for your minor version.
If required, perform incremental minor version updates
If you plan to upgrade to a version that is more than one minor version greater than your current version, such as to version 1.19 from 1.17 or older, you must upgrade incrementally. For example, you must first use the upgrade guide in the v1.18.x documentation set to upgrade from 1.17 to 1.18, and then follow the upgrade guide in the v1.19.x documentation set to upgrade from 1.18 to 1.19.
Upgrade dependencies
Check that your underlying infrastructure platform, such as Kubernetes, and other dependencies run a version that is supported for 1.19.
- Review the supported versions for dependencies such as Kubernetes, Istio, Helm, and more.
- Compare the supported versions against the versions you currently use.
- If necessary, upgrade your dependencies, such as consulting your cluster infrastructure provider to upgrade the version of Kubernetes that your cluster runs.
Consider settings to avoid downtime
You might deploy Gloo Gateway in Kubernetes environments that use the Kubernetes load balancer, or in non-Kubernetes environments. Depending on your setup, you can take additional steps to avoid downtime during the upgrade process.
- Kubernetes: Enable Envoy readiness and liveness probes during the upgrade. When these probes are set, Kubernetes sends requests only to the healthy Envoy proxy during the upgrade process, which helps to prevent potential downtime. The probes are not enabled in default installations because they can lead to timeouts or other poor getting started experiences.
- Non-Kubernetes: Configure health checks on Envoy. Then, configure your load balancer to leverage these health checks, so that requests stop going to Envoy when it begins draining connections.
Review version 1.19 changes
Review the following changes made to Gloo Gateway in version 1.19. For some changes, you might be required to complete additional steps during the upgrade process.
Breaking changes
Envoy version upgrade
The Envoy dependency in Gloo Gateway 1.19 was upgraded from 1.31.x to 1.33.x. This change includes the following upstream breaking changes. For more information about these changes, see the Envoy changelog documentation.
- Trust internal addresses: Envoy sanitizes certain headers and statistics for non-trusted addresses. By default, Envoy is configured to trust internal addresses that adhere to the RFC1918 standard. However, in Envoy 1.33, this behavior is changed. RFC1918 addresses must now be added to the HCM
internal_address_config
for Envoy to trust them. For example, if you have tooling, such as probes on your private network, make sure to include these IP addresses or CIDR ranges in the internal address config. To try out or enable this change, you can set theenvoy.reloadable_features.explicit_internal_address_config
runtime guard totrue
. For more information, see the related pull request in Envoy. - Access log handlers: Access log handlers that are added by filters are now evaluated before access log handlers that are configured in the
access_log
configuration. To disable this behavior, you can set theenvoy.reloadable_features.filter_access_loggers_first
runtime guard flag tofalse
. - Cluster name change in Kuberetes Gateway API: When using the Kubernetes Gateway API alongside Gloo Edge APIs and you route to Kubernetes Services or Upstreams, the Envoy cluster name format is changed to extract more details about the service. The new format uses underscores to list service details, such as
upstreamName_upstreamNs_svcNs_svcName_svcPort
. If you enabled the Kubernetes Gateway API integration withkubeGateway.enabled=true
, both the Gloo Edge and Kubernetes Gateway API proxies use the same format for these cluster names. - Extproc tracing changes: In previous releases, tracing spans that were generated by the extProc filter were sampled by default. Now, these traces are not automatically sampled. Instead, the tracing decision is inherited from the parent span.
- Opencensus tracing extension removed: Support for the Opencensus tracing extension is removed in Envoy 1.33.
- Opentracing extension removed: Support for the Opentracing extension is removed in Envoy 1.32. For more information, see the related Envoy issue.
New features
Set authority header for gRPC OpenTelemetry collectors
When referencing a gRPC OpenTelemetry collector in your Gateway, Gloo Gateway automatically generates an Envoy configuration that sets the cluster name as the :authority
pseudo-header. If your collector expects a different :authority
header, you can specify that by setting the spec.httpGateway.options.httpConnectionManagerSettings.tracing.openTelemetryConfig.grpcService.authority
value on your Gateway as shown in the following example.
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
labels:
app: gloo
app.kubernetes.io/name: gateway-proxy-tracing-authority
name: gateway-proxy-tracing-authority
spec:
bindAddress: '::'
bindPort: 18082
proxyNames:
- gateway-proxy
httpGateway:
virtualServiceSelector:
gateway-type: tracing
options:
httpConnectionManagerSettings:
tracing:
openTelemetryConfig:
collectorUpstreamRef:
name: opentelemetry-collector
namespace: default
grpcService:
authority: my-authority
Log filter state in gRPC access logs
You can enable logging of the filter state when performing gRPC access logging. The filter state logger calls the FilterState::Object::serializeAsProto
to serialize the filter state object.
You can enable the filter in your Helm values file or a Gateway resource directly.
The following example adds the modsecurity object from a WAF policy to the filter state object in the access log.
gloo:
accessLogger:
enabled: true
image:
registry: quay.io/solo-io
tag: 1.0.0-ci1
gatewayProxies:
gatewayProxy:
gatewaySettings:
customHttpGateway:
options:
waf:
auditLogging:
action: ALWAYS
location: FILTER_STATE
customInterventionMessage: 'ModSecurity intervention! Custom message details here..'
ruleSets:
- ruleStr: |
# Turn rule engine on
SecRuleEngine On
SecRule REQUEST_HEADERS:User-Agent "scammer" "deny,status:403,id:107,phase:1,msg:'blocked scammer'"
accessLoggingService:
accessLog:
- grpcService:
logName: example
staticClusterName: access_log_cluster
filterStateObjectsToLog:
- io.solo.modsecurity.audit_log
The following example adds the modsecurity object from a WAF policy to the filter state object in the access log.
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
name: gateway-proxy
namespace: gloo-system
spec:
bindAddress: '::'
bindPort: 8080
proxyNames:
- gateway-proxy
httpGateway:
options:
waf:
customInterventionMessage: 'ModSecurity intervention! Custom message details here..'
ruleSets:
- ruleStr: |
# Turn rule engine on
SecRuleEngine On
SecRule REQUEST_HEADERS:User-Agent "scammer" "deny,status:403,id:107,phase:1,msg:'blocked scammer'"
useProxyProto: false
options:
accessLoggingService:
accessLog:
- grpcService:
logName: example
staticClusterName: access_log_cluster
filterStateObjectsToLog:
- io.solo.modsecurity.audit_log
Example access log output:
"filter_state_objects":{"io.solo.modsecurity.audit_log":{"type_url":"type.googleapis.com/google.protobuf.StringValue","value":"Ct0DLS0tQS0tClsxMC9NYXIvMjAyNToxNDoyNTowMSArMDAwMF0gMTc0MTYxNjcwMTQ1LjkwMTMzMCAxMjcuMC4wLjEgMzg5NTQgIDAKLS0tQi0tCkdFVCAvYWxsLXBldHMgSFRUUC8xLjEKOm1ldGhvZDogR0VUCmhvc3Q6IGxvY2FsaG9zdDo4MDgwCjpwYXRoOiAvYWxsLXBldHMKOmF1dGhvcml0eTogbG9jYWxob3N0OjgwODAKeC1mb3J3YXJkZWQtcHJvdG86IGh0dHAKOnNjaGVtZTogaHR0cAp1c2VyLWFnZW50OiBjdXJsLzguMTEuMAphY2NlcHQ6ICovKgp4LXJlcXVlc3QtaWQ6IDUwZTBjZTVmLTk0NjktNGQ3NS1hMzUxLWU4MmI0ODFlYTFmYgoKLS0tRi0tCkhUVFAvMS4xIDIwMAo6c3RhdHVzOiAyMDAKY29udGVudC10eXBlOiBhcHBsaWNhdGlvbi94bWwKZGF0ZTogTW9uLCAxMCBNYXIgMjAyNSAxNDoyNTowMSBHTVQKY29udGVudC1sZW5ndGg6IDg2CngtZW52b3ktdXBzdHJlYW0tc2VydmljZS10aW1lOiAyCgotLS1ILS0KCi0tLVotLQoK"}}}
For more information, see the Access logging API.
Match conditions on validation webhook
You can now specify match conditions on the Gloo Gateway or Kubernetes validating admission webhook level to filter the resources that you want to include or exclude from validation. Match conditions are written in CEL (Common Expression Language).
For example to exclude a secret with the label “foo” from being validated with the Kubernetes webhook, add the following values to your Helm values file.
gateway:
validation:
kubeCoreFailurePolicy: Fail # For "strict" validation mode, fail the validation if webhook server is not available
kubeCoreMatchConditions:
- name: 'not-a-secret-or-secret-with-foo-label-key'
expression: '!(request.kind.kind == "Secret" && "labels" in object.metadata && "foo" in object.metadata.labels)'
To exclude these secrets from the Gloo Gateway validation webhook, add the following values.
gateway:
validation:
FailurePolicy: Fail # For "strict" validation mode, fail the validation if webhook server is not available
matchConditions:
- name: 'not-a-secret-or-secret-with-foo-label-key'
expression: '!(request.kind.kind == "Secret" && "labels" in object.metadata && "foo" in object.metadata.labels)'
For more information, see Exclude resources from validation.
Send gateway access logs to OTel collector
You can configure your Gateway to send access logs to an OpenTelemetry collector as shown in the following example.
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
name: gateway-proxy
namespace: gloo-system
spec:
bindAddress: '::'
bindPort: 8080
httpGateway: {}
proxyNames:
- gateway-proxy
useProxyProto: false
options:
accessLoggingService:
accessLog:
- openTelemetryService:
logName: example
collector:
endpoint: otel-collector.default.svc.cluster.local:4317
For more information and additional settings, see the Access log API.
Kubernetes 1.32 support
Starting in version 1.19.0, Gloo Gateway can now run on Kubernetes 1.32. For more information about supported Kubernetes, Envoy, and Istio versions, see Supported versions.
Istio 1.25 support
Starting in version 1.19.0, Gloo Gateway can now run with Istio 1.25. For more information about supported Kubernetes, Envoy, and Istio versions, see Supported versions.
New common category for CRs
A new common category was added to all Gloo Gateway custom resources. This common category allows you to easily list all Gloo Gateway custom resources in your environment.
To list all Gloo Gateway resources across all namespaces:.
kubectl get gloo-gateway -A
To list all enterprise custom resources only:
kubectl get solo-io -A
Changelogs
Check the changelogs for the type of Gloo Gateway deployment that you have. Focus especially on any Breaking Changes that might require a different upgrade procedure. For Gloo Gateway Enterprise, you might also review the open source changelogs because most of the proto definitions are open source.
- Open Source changelogs
- Enterprise changelogs: Keep in mind that Gloo Gateway Enterprise pulls in Gloo Gateway Open Source as a dependency. Although the major and minor version numbers are the same for open source and enterprise, their patch versions often differ. For example, open source might use version
x.y.a
but enterprise uses versionx.y.b
. If you are unfamiliar with these versioning concepts, see Semantic versioning. Because of the differing patch versions, you might notice different output when checking your version withglooctl version
. For example, your API server might run Gloo Gateway Enterprise version 1.19.1, which pulls in Gloo Gateway Open Source version 1.19.1 as a dependency.~ > glooctl version Client: {"version":"1.19.1"} Server: {"type":"Gateway","enterprise":true,"kubernetes":...,{"Tag":"1.19.1","Name":"grpcserver-ee","Registry":"quay.io/solo-io"},...,{"Tag":"1.19.1","Name":"discovery","Registry":"quay.io/solo-io"},...}
You can use the changelogs' built-in comparison tool to compare between your current version and the version that you want to upgrade to.
CLI changes
You must upgrade glooctl
before you upgrade Gloo Gateway. Because glooctl
can create resources in your cluster, such as with glooctl add route
, you might have errors in Gloo Gateway if you create resources with an older version of glooctl
.
Review the following summary of important new, deprecated, or removed CLI options. For full details, see the changelogs.
New CLI commands or options:
glooctl debug
andglooctl debug yaml
: Collect Kubernetes, Gloo Gateway controller, and Envoy information from your environment, such as logs, YAML manifests, metrics, and snapshots. This information can be used to debug issues in your environment or to provide this information to the Solo.io support team.glooctl gateway api convert
: Use this command to convert Gloo Edge APIs to Kubernetes Gateway API YAML files so that you can preview and run the migration to Gloo Gateway with the Kubernetes Gateway API.
Frequently-asked questions
Review the following frequently-asked questions about the upgrade process. If you still aren’t sure about the version upgrade impact, or if your use case doesn’t quite fit the standard upgrade path, feel free to post in the #gloo
or #gloo-enterprise
channels of our public Slack.
How do I upgrade Gloo Gateway in testing or sandbox environments?
If downtime is not a concern for your use case, you can follow the Quick upgrade guide to update your Gloo Gateway installation.
Note that for sandbox or exploratory environments, the easiest way to upgrade is to uninstall Gloo Gateway by running glooctl uninstall --all
. Then, re-install Gloo Gateway at the desired version by the following one of the installation guides.
How do I upgrade Gloo Gateway in a production environment, where downtime is unacceptable?
The basic helm upgrade
process is not suitable for environments in which downtime is unacceptable. Instead, you can follow the Canary upgrade guide to deploy multiple version of Gloo Gateway to your cluster, and test the upgrade version before uninstalling the existing version.
Additionally, you might need to take steps to account for other factors such as Gloo Gateway version changes, probe configurations, and external infrastructure like the load balancer that Gloo Gateway uses. Consider setting up liveness probes and healthchecks in your environment.
What happens to my Gloo Gateway CRs during an upgrade? How do I handle breaking changes?
A typical upgrade of Gloo Gateway across minor versions should not cause disruptions to the existing Gloo Gateway state. In the case of a breaking change, Solo will communicate through the upgrade guides, changelogs, or other channels if you must make a specific adjustment to perform the upgrade. Note that you can always use the glooctl debug yaml
command to download the current Gloo Gateway state to one large YAML manifest.
Is the upgrade procedure different if I am not a cluster administrator?
If you are not an administrator of your cluster, you might be unable to create custom resource definitions (CRDs) and other cluster-scoped resources, such as cluster roles and cluster role bindings. If you encounter an error related to these resources, you can disable their creation by including the following setting in your Helm values:
global:
glooRbac:
create: false
Otherwise, you can try performing an installation of Gloo Gateway that is scoped to a single namespace by including the following setting in your Helm values:
global:
glooRbac:
namespaced: true
Why do I get an error about re-creating CRDs when upgrading using helm install
or helm upgrade
?
Helm v2 does not manage CRDs well, and is not supported in Gloo Gateway. Upgrade to Helm v3, delete the CRDs, and try again.
Why do I get an error about a gateway-certgen
job?
The upgrade creates a Kubernetes Job named gateway-certgen
to generate a certificate for the validation webhook. The job contains the ttlSecondsAfterFinished
value so that the cluster cleans up the job automatically, but because this setting is still in Alpha, your cluster might ignore this value. In this case, you might have an issue while upgrading in which the upgrade attempts to change the gateway-certgen
job, but the change fails because the job is immutable. To fix this issue, you can delete the job, which already completed, and re-apply the upgrade.