Navigation :

Upgrade

Upgrade minor and patch versions for the Gloo Gateway management server, agent, and gateway proxy.

During the upgrade, the data plane continues to run, but you might not be able to modify the configurations through the management plane. Because zero downtime is not guaranteed, try testing the upgrade in a staging environment before upgrading your production environment.

Before you begin

Review the changelog for differences from the previous minor version. Focus especially on any Breaking Changes that might require a different upgrade procedure. For example, take note of the following changes in version 2.5 or later that you might need to address before upgrading.
- Default Gloo Platform add-ons namespace removed: In previous releases, all add-ons were automatically installed to the gloo-mesh-addons namespace unless you specified a different namespace during the Gloo Mesh Enterprise installation. Starting with release v2.5.0, this default value is removed. If no value is set in the common.addonsNamespace Helm field, your add-ons are now deployed to the namespace that the Helm release is installed to. To avoid disruptions or downtime for your add-on components, such as a rate limit server, set the namespace you want your add-ons to be installed to in the common.addonsNamespace field of your Helm values file.
Check that your underlying Kubernetes platform runs a supported version for the Gloo version that you want to upgrade to.
1. Review the supported versions.
2. Compare the supported version against the version of Kubernetes that you run in your clusters.
3. If necessary, upgrade Kubernetes. Consult your cluster infrastructure provider.
Set the Gloo Gateway version that you want to upgrade to as an environment variable. The latest version is used as an example. You can find other versions in the Changelog documentation. Append ‘-fips’ for a FIPS-compliant image, such as ‘2.5.0-beta2-fips’. Do not include v before the version number.
```
export UPGRADE_VERSION=2.5.0-beta2
```

Looking to update certain Helm chart values but not the version? Skip to step 2.

Step 1: Upgrade Gloo CRDs

Update the Helm repository for Gloo Platform.

helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts
helm repo update

Apply the Gloo custom resource definitions (CRDs) for the target version by upgrading your gloo-platform-crds Helm release.

Single cluster
Multicluster

helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \
   --namespace=gloo-mesh \
   --version=$UPGRADE_VERSION

Upgrade your gloo-platform-crds Helm release in the management cluster.

helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \
   --kube-context $MGMT_CONTEXT \
   --namespace=gloo-mesh \
   --version=$UPGRADE_VERSION

Upgrade your gloo-platform-crds Helm release in each workload cluster. Remember to change the context for each workload cluster that you upgrade.

helm upgrade gloo-platform-crds gloo-platform/gloo-platform-crds \
   --kube-context $REMOTE_CONTEXT \
   --namespace=gloo-mesh \
   --version=$UPGRADE_VERSION

Step 2: Get your Helm chart values

As part of the upgrade, you can update or reuse the Helm chart values for your Gloo management server, agent, and any add-ons that you might have such as rate limiting and external authentication services.

Get the Helm values file for your current version.
- Single cluster
- Multicluster
1. Get your current values. Note that if you migrated from the legacy Helm charts, your Helm release might be named gloo-mgmt or gloo-mesh-enterprise instead.
  helm get values gloo-platform -n gloo-mesh > gloo-gateway-single.yaml open gloo-gateway-single.yaml
2. Delete the first line that contains USER-SUPPLIED VALUES:, and save the file.
3. Optional: If you maintain a separate gloo-agent-addons Helm release, get the values for that Helm release too, and delete the first line that contains USER-SUPPLIED VALUES:.
  helm get values gloo-agent-addons -n gloo-mesh-addons > gloo-agent-addons.yaml open gloo-agent-addons.yaml
1. Get your current values for the management cluster. Note that if you migrated from the legacy Helm charts, your Helm release might be named gloo-mgmt or gloo-mesh-enterprise instead.
  helm get values gloo-platform -n gloo-mesh --kube-context $MGMT_CONTEXT > mgmt-server.yaml open mgmt-server.yaml
2. Delete the first line that contains USER-SUPPLIED VALUES:, and save the file.
3. Get your current values for the workload clusters. Note that if you migrated from the legacy Helm charts, your Helm release might be named gloo-agent or gloo-mesh-agent instead.
  helm get values gloo-platform -n gloo-mesh --kube-context $REMOTE_CONTEXT > agent.yaml open agent.yaml
4. Delete the first line that contains USER-SUPPLIED VALUES:, and save the file.
5. Optional: If you maintain a separate gloo-agent-addons Helm release, get the values for that Helm release too, and delete the first line that contains USER-SUPPLIED VALUES:.
  helm get values gloo-agent-addons -n gloo-mesh-addons --kube-context $REMOTE_CONTEXT > gloo-agent-addons.yaml open gloo-agent-addons.yaml
Compare your current Helm chart values with the version that you want to upgrade to. You can get a values file for the upgrade version with the helm show values command.
```
helm show values gloo-platform/gloo-platform --version $UPGRADE_VERSION > all-values.yaml
```
Review the changelog for any Helm Changes that might require modifications to your Helm chart.
- Default Gloo Platform add-ons namespace removed: In previous releases, all add-ons were automatically installed to the gloo-mesh-addons namespace unless you specified a different namespace during the Gloo Mesh Enterprise installation. Starting with release v2.5.0, this default value is removed. If no value is set in the common.addonsNamespace Helm field, your add-ons are now deployed to the namespace that the Helm release is installed to. To avoid disruptions or downtime for your add-on components, such as a rate limit server, set the namespace you want your add-ons to be installed to in the common.addonsNamespace field of your Helm values file.
Edit the Helm values file or prepare the --set flags to make any changes that you want. If you do not want to use certain settings, comment them out.

Updating values in the istioInstallations section? See Upgrade managed gateway proxies for special instructions.

Step 3: Upgrade and verify the Helm installation

Upgrade the Gloo Gateway Helm installation. Make sure to include your Helm values when you upgrade either as a configuration file in the --values flag or with --set flags. Otherwise, any previous custom values that you set might be overwritten. In single cluster setups, this might mean that your Gloo agent and ingress gateways are removed.
- Single cluster
- Multicluster
1. Upgrade your Helm release. Change the release name as needed.
  helm upgrade gloo-platform gloo-platform/gloo-platform \ --namespace gloo-mesh \ -f gloo-gateway-single.yaml \ --version $UPGRADE_VERSION
2. Optional: If you migrated from the legacy Helm charts and maintained a separate gloo-agent-addons Helm release during the migration, upgrade that Helm release too.
  helm upgrade gloo-agent-addons gloo-platform/gloo-platform \ --namespace gloo-mesh-addons \ -f gloo-agent-addons.yaml \ --version $UPGRADE_VERSION
In multicluster setups, you must always upgrade the Gloo management server before upgrading the Gloo agent to avoid unexpected behavior. Note that only n-1 minor version skew is supported between the management server and the agent. For more information, see the Skew policy.
1. Upgrade your Helm release in the management cluster. Change the release name as needed.
  helm upgrade gloo-platform gloo-platform/gloo-platform \ --kube-context $MGMT_CONTEXT \ --namespace gloo-mesh \ -f mgmt-server.yaml \ --version $UPGRADE_VERSION
2. Upgrade your Helm release in each workload cluster. Change the release name as needed. Be sure to update the cluster context for each workload cluster that you repeat this command for.
  helm upgrade gloo-platform gloo-platform/gloo-platform \ --kube-context $REMOTE_CONTEXT \ --namespace gloo-mesh \ -f agent.yaml \ --version $UPGRADE_VERSION
3. Optional: If you migrated from the legacy Helm charts and maintained a separate gloo-agent-addons Helm release during the migration, upgrade that Helm release in each workload cluster too. Be sure to update the cluster context for each workload cluster that you repeat this command for.
  helm upgrade gloo-agent-addons gloo-platform/gloo-platform \ --kube-context $REMOTE_CONTEXT \ --namespace gloo-mesh-addons \ -f gloo-agent-addons.yaml \ --version $UPGRADE_VERSION
Optional: Check that the Gloo management and agent resources are connected.
```
meshctl check
```

Confirm that the server components such as gloo-mesh-mgmt-server run the version that you upgraded to.

meshctl version

Example output:

   "server": [
   {
     "Namespace": "gloo-mesh",
     "components": [
       {
         "componentName": "gloo-mesh-mgmt-server",
         "images": [
            {
             "name": "gloo-mesh-mgmt-server",
             "domain": "gcr.io",
             "path": "gloo-mesh-mgmt-server",
             "version": "2.5.0-beta2"
           }
         ]
       },

Multicluster setups only: Confirm that the agent components such as gloo-mesh-agent run the version that you upgraded to.

meshctl version --kubecontext ${REMOTE_CONTEXT}

Example output:

   {
         "componentName": "gloo-mesh-agent",
         "images": [
           {
             "name": "gloo-mesh-agent",
             "domain": "gcr.io",
             "path": "gloo-mesh/gloo-mesh-agent",
             "version": "2.5.0-beta2"
           }
         ]
       },

Next steps

Now that you upgraded Gloo, you must upgrade your meshctl CLI to the matching version. Depending on the Gloo version support, you might also want to upgrade Kubernetes in your clusters.

Upgrade the meshctl CLI to the same version of Gloo.
Optional: If the new version of Gloo supports a more recent version of Kubernetes, you can upgrade Kubernetes on your cluster. For more information, consult your cluster infrastructure provider.

Upgrade managed gateway proxies

During step 2 of the Helm upgrade process, you might make changes to the istioInstallations section of your Helm values file to update your Istio control plane and gateway proxies. Depending on the type of change, you apply updates to the installations in one of the following ways:

Revisioned canary upgrade (recommended)
In-place upgrade
Testing only: Manually replacing the GatewayLifecycleManager CR

Istio versions 1.17 and later and Gloo Platform 2.4 and later do not support the Gloo legacy metrics pipeline. If you run the legacy metrics pipeline, be sure that you set up the Gloo OpenTelemetry (OTel) pipeline instead in your new or existing Gloo Gateway installation.

Revisioned canary upgrades (recommended)

In a canary upgrade, you install another Istio installation (canary) alongside your active installation. Each installation is revisioned so that you can easily identify and verify the separate settings and resources for each installation. Note that during a canary upgrade, the validating admissions webhook is enabled only for the canary installation to prevent issues that occur when multiple webhooks are enabled.

Perform a canary upgrade when you change one of the following fields:

istioOperatorSpec.tag minor version
istioOperatorSpec.hub repository, such as switching to the Solo Istio repository for the minor version you want to upgrade to
components, profile, values, or namespace in the istioOperatorSpec

To perform a canary upgrade:

OpenShift only: Elevate the permissions of the service account that will be created for the new revision's operator project. This permission allows the ingress gateway proxy to make use of a user ID that is normally restricted by OpenShift. Replace the revision with the revision you plan to use.
```
oc adm policy add-scc-to-group anyuid system:serviceaccounts:gm-iop-1-19-3
```

Follow the steps in this guide to perform a regular upgrade of your Gloo Gateway installation. When you edit the istioInstallations.controlPlane and istioInstallations.northSouthGateways sections of your Helm values file, add another installation entry for the canary revision, and leave the entry your your current installation as-is. For the canary revision, be sure to set defaultRevision and activeGateway to false so that only the existing revisions continue to run.

For example, you might add the following installation entries for the Istio control plane and ingress gateway alongside your existing entries. If you have a Gloo Gateway license, you might also have entries for the ingress gateway proxy in the nothSouthGateways section too.

istioInstallations:
    controlPlane:
        enabled: true
        installations:
            # EXISTING revision
            - clusters:
                  # Keep this field set to TRUE
                - defaultRevision: true
                  name: cluster1
                  trustDomain: ""
              istioOperatorSpec:
                hub: $REPO
                tag: 1.18.3-solo
                profile: minimal
                namespace: istio-system
                ...
              revision: 1-18-3
            # NEW revision
            - clusters:
                  # Set this field to FALSE
                - defaultRevision: false
                  name: cluster1
                  trustDomain: ""
              istioOperatorSpec:
                hub: $REPO
                tag: 1.19.3-solo
                profile: minimal
                namespace: istio-system
                ...
              revision: 1-19-3
    eastWestGateways: null
    enabled: true
    northSouthGateways:
        - enabled: true
          installations:
            # EXISTING revision
            - clusters:
                  # Keep this field set to TRUE
                - activeGateway: true
                  name: cluster1
                  name: 
                  trustDomain: ""
              gatewayRevision: 1-18-3
              istioOperatorSpec:
                hub: $REPO
                tag: 1.18.3-solo
                profile: empty
                namespace: gloo-mesh-gateways
                ...
            # NEW revision
            - clusters:
                  # Set this field to FALSE
                - activeGateway: false
                  name: cluster1
                  name: 
                  trustDomain: ""
              gatewayRevision: 1-19-3
              istioOperatorSpec:
                hub: $REPO
                tag: 1.19.3-solo
                profile: empty
                namespace: gloo-mesh-gateways
                ...
          name: istio-ingressgateway

Updating the minor version of Istio? In your canary revision section, be sure to update both the repo key in the hub field, and the Istio version in the tag field. You can get the repo key for the Istio version that you want to install from the Istio images built by Solo.io support article.

For most use cases, you can set the revision and the gatewayRevision to the same version. However, gateway installations can point to any istiod control plane revision by using the controlPlaneRevision field. For simplicity, if you do not specify controlPlaneRevision, the gateway installation uses a control plane with the same revision as itself.

After you apply the Helm upgrade with your updated values file, verify that Istio resources for the canary installation are created. For example, if you updated the Istio minor version to 1-19-3, verify that resources are created in the gm-iop-1-19-3 namespace, and that resources for 1-19-3 are created alongside the existing resources for the previous version in the istio-system and gloo-mesh-gateways namespaces. Note that the gateway load balancers for the canary revision contain the revision in the name, such as istio-ingressgateway-1-19-3.
```
kubectl get all -n gm-iop-1-19-3
kubectl get all -n istio-system
kubectl get all -n gloo-mesh-gateways
```
Running into issues or seeing a stuck canary upgrade? In testing environments, you can clear your configuration by manually replacing the GatewayLifecycleManager CR.

After performing any necessary testing, switch to the new Istio control plane and ingress gateway revisions.

Get your Helm values file. Change the release name as needed.

helm get values gloo-platform -n gloo-mesh > gloo-gateway-single.yaml
open gloo-gateway-single.yaml

Change defaultRevision and activeGateway to false for the old revision and to true for the new revision.

New load balancers are created for the canary gateways. To instead change the control plane revision in use by the existing gateway load balancers, you can set the istio.io/rev label on the gateway deployment, which triggers a rolling restart.

istioInstallations:
    controlPlane:
        enabled: true
        installations:
            # OLD revision
            - clusters:
                  # Set this field to FALSE
                - defaultRevision: false
                  name: cluster1
                  trustDomain: ""
              istioOperatorSpec:
                hub: $REPO
                tag: 1.18.3-solo
                profile: minimal
                namespace: istio-system
                ...
              revision: 1-18-3
            # NEW revision
            - clusters:
                  # Set this field to TRUE
                - defaultRevision: true
                  name: cluster1
                  trustDomain: ""
              istioOperatorSpec:
                hub: $REPO
                tag: 1.19.3-solo
                profile: minimal
                namespace: istio-system
                ...
              revision: 1-19-3
    eastWestGateways: null
    enabled: true
    northSouthGateways:
        - enabled: true
          installations:
            # OLD revision
            - clusters:
                  # Set this field to FALSE
                - activeGateway: false
                  name: cluster1
                  name: 
                  trustDomain: ""
              gatewayRevision: 1-18-3
              istioOperatorSpec:
                hub: $REPO
                tag: 1.18.3-solo
                profile: empty
                namespace: gloo-mesh-gateways
                ...
            # NEW revision
            - clusters:
                  # Set this field to TRUE
                - activeGateway: true
                  name: cluster1
                  name: 
                  trustDomain: ""
              gatewayRevision: 1-19-3
              istioOperatorSpec:
                hub: $REPO
                tag: 1.19.3-solo
                profile: empty
                namespace: gloo-mesh-gateways
                ...
          name: istio-ingressgateway

Upgrade your Helm release. Change the release name as needed.

helm upgrade gloo-platform gloo-platform/gloo-platform \
   --namespace gloo-mesh \
   -f gloo-gateway-single.yaml \
   --version $UPGRADE_VERSION

After your Helm upgrade completes, verify that the active gateways for the new revision are created, which do not have the revision appended to the name. Note that gateways for the inactive revision that you previously ran also exist in the namespace, in the case that a rollback is required.
```
kubectl get all -n gloo-mesh-gateways
```
Example output, in which the active gateway (istio-ingressgateway) for the new revision and inactive gateway (such as istio-ingressgateway-1-18-3) for the old revision are created:
```
NAME                            TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)     
istio-ingressgateway            LoadBalancer   10.44.4.140   34.150.235.221   15021:31321/TCP,80:32525/TCP,443:31826/TCP   48s                                                 AGE
istio-ingressgateway-1-18-3       LoadBalancer   10.56.15.36    34.145.163.61   15021:31936/TCP,80:30196/TCP,443:32286/TCP,15443:31851/TCP   45s
```
To uninstall the previous installations, or if you need to uninstall the canary installations, you can edit your Helm values file to remove the revision entries from the istioInstallations.controlPlane.installations and istioInstallations.northSouthGateways.installations lists. Then, upgrade your Gloo Gateway Helm release with your updated values file.
If you also use Gloo Mesh Enterprise alongside Gloo Gateway, see step 6 in the Gloo Mesh upgrade documentation to upgrade your workloads’ Istio sidecars.

In-place upgrades

In an in-place upgrade, Gloo upgrades your existing control plane or gateway installations. Note that in production environments, canary upgrades are recommended for updating the minor version. However, you might want to use in-place upgrades for patch versions or changes within the same minor version. Be sure to test in-place upgrades in development or staging environments first.

In-place upgrades behave differently based on whether your installation is revisionless or revisioned.

Revisionless installations (testing only): If your testing-only installation is revisionless (your settings omit the revision and gatewayRevision fields), in-place upgrades are triggered when you apply changes to any field in the istioInstallations section.

Revisioned installations: If your installation is revisioned, in-place upgrades are triggered only when you apply changes to one of the following fields in the istioInstallations section. Otherwise, a canary upgrade is required.

istioOperatorSpec.tag patch version
- In-place upgrades are not supported for downgrading the patch version.
- In-place upgrades are not supported if you do not already specify a tag value, such as if you want to switch from the auto setting to a specific version. This is because you must also specify hub and revision values, which require a canary upgrade.
istioOperatorSpec.meshConfig values

To trigger an in-place upgrade:

Follow the steps in this guide to perform a regular upgrade of your Gloo Gateway installation and include your Istio changes in your Helm values file. For example, in a single-cluster setup, you might edit your Helm values file to update the patch version of Istio in the istioInstallations.controlPlane.installations.istioOperatorSpec.tag and istioInstallations.northSouthGateways.installations.istioOperatorSpec.tag fields. After you apply the updates in your Helm upgrade of the gloo-platform chart, Gloo starts an in-place upgrade of the istiod control plane and the ingress gateway proxy.
After your Helm upgrade completes, restart your gateway proxy pods. For example, you might use the following command to rollout a restart of the istio-ingressgateway-1-19-3 deployment.
```
kubectl rollout restart -n gloo-mesh-gateways deployment/istio-ingressgateway-1-19-3
```

Verify that your Istio resources are updated.

kubectl get all -n gm-iop-1-19-3
kubectl get all -n istio-system
kubectl get all -n gloo-mesh-gateways

Testing only: Manually replacing the GatewayLifecycleManager CR

In testing or demo setups, you can quickly upgrade your managed gateway proxies by manually deleting the GatewayLifecycleManager CR, and upgrading your Gloo Gateway installation with your updated gateway values in your Helm values file. Note that you can also use this method to clear your managed gateway configurations if a canary upgrade becomes stuck.

This method is supported only for testing scenarios, because your ingress gateway proxies are temporarily removed in this process.

Get the name of your GatewayLifecycleManager resource. Typically, this resource is named istio-ingressgateway.
```
kubectl get GatewayLifecycleManager -A
```

Delete the resource.

kubectl delete GatewayLifecycleManager istio-ingressgateway -n gloo-mesh

Verify that your gateway proxy is removed. It might take a few minutes for the service to delete.
```
kubectl get all -n gloo-mesh-gateways
```
Optional: If you also need to make changes to your Istio control plane, clear the istiod configuration.
1. Get the name of your IstioLifecycleManager resource. Typically, this resource is named gloo-platform.
```
kubectl get IstioLifecycleManager -A
```
2. Delete the resource.
```
kubectl delete IstioLifecycleManager gloo-platform -n gloo-mesh
```
3. Verify that your istiod control plane is removed.
```
kubectl get all -n istio-system
```
Follow the steps in this guide to perform a regular upgrade of your Gloo Gateway installation and include your Istio changes in your Helm values file. After you apply the updates in your Helm upgrade of the gloo-platform chart, Gloo re-creates the ingress gateway proxy, and if applicable, the istiod control plane.

After your Helm upgrade completes, verify that your Istio resources are re-created.

# Change the revision as needed
kubectl get all -n gm-iop-1-19-3
kubectl get all -n istio-system
kubectl get all -n gloo-mesh-gateways

Update your Gloo license

Before your Gloo license expires, you can update the license by patching the license key secret. If you use Gloo Gateway along with other Gloo products such as Gloo Mesh and Gloo Network, you can also update those licenses.

For example, if you notice that your Gloo control plane deployments are in a crash loop, your Gloo license might be expired. You can check the status of your license with the following command:

meshctl check

Example output for an expired license:

🔴 License status

ERROR A license is expired. To update, see https://docs.solo.io/gloo-gateway/main/setup/prepare/licensing/#update-licenses
INFO  gloo-gateway enterprise license expiration is 06 Feb 23 13:05 CDT
INFO  Valid GraphQL license module found

To update your license key in your Gloo installation:

Get a new Gloo license key by contacting your account representative. If you use Gloo Gateway along with other Gloo products such as Gloo Mesh and Gloo Network, make sure to ask for up-to-date license keys for all your products.

Save the new license key as an environment variable.

Gateway
Mesh
Network

export GLOO_GATEWAY_LICENSE_KEY=<new-key-string>

export GLOO_MESH_LICENSE_KEY=<new-key-string>

export GLOO_NETWORK_LICENSE_KEY=<new-key-string>

Update the license secret to use the new Gloo Gateway license key.

Gateway
Mesh
Network

kubectl -n gloo-mesh patch secret license-keys --patch='{"stringData": { "gloo-gateway-license-key": "GLOO_GATEWAY_LICENSE_KEY" }}' -v=1

kubectl -n gloo-mesh patch secret license-keys --patch='{"stringData": { "gloo-mesh-license-key": "$GLOO_MESH_LICENSE_KEY" }}' -v=1

kubectl -n gloo-mesh patch secret license-keys --patch='{"stringData": { "gloo-network-license-key": "$GLOO_NETWORK_LICENSE_KEY" }}' -v=1

Optional: If your license expired and the management server pods are in a crash loop, restart the management server pods. If you updated the license before expiration, skip this step.
```
kubectl rollout restart -n gloo-mesh deployment/gloo-mesh-mgmt-server
```

Verify that your license check is now valid, and no errors are reported.

meshctl check

Example output:

🟢 License status

INFO  gloo-gateway enterprise license expiration is 25 Aug 24 10:38 CDT
INFO  Valid GraphQL license module found