You can use this guide to upgrade the Gloo version of your Gloo components, such as the management server and agents, or to apply changes to the components’ configuration settings.

Considerations

Consider the following rules before you plan your Gloo Mesh Gateway upgrade.

Testing upgrades

During the upgrade, the data plane continues to run, but you might not be able to modify the configurations through the management plane. Because zero downtime is not guaranteed, try testing the upgrade in a staging environment before upgrading your production environment.

Patch and minor versions

Patch version upgrades:
You can skip patch versions within the same minor release. For example, you can upgrade from version 2.6.0 to 2.6.5 directly, and skip the patch versions in between.

Minor version upgrades:

  • Always upgrade to the latest patch version of the target minor release. For example, if you want to upgrade from version 2.5.11 to 2.6.x, and 2.6.5 is the latest patch version, upgrade to that version and skip any previous patch versions for that minor release. Do not upgrade to a lower patch version, such as 2.6.0, 2.6.1, and so on.
  • Do not skip minor versions during your upgrade. Upgrade minor release versions one at a time. For example, if you want to upgrade from 2.3.x to 2.5.x, you must first upgrade to the latest patch version of the 2.4 minor release. After you upgrade to 2.4.x, you can then plan your upgrade to the latest patch version of the 2.5.x release.

Multicluster only: Version skew policy for management and remote clusters

Plan to always upgrade your Gloo management server and agents to the same target version. Always upgrade the Gloo management server first. Then, roll out the upgrade to the Gloo agents in your workload clusters. During this upgrade process, your management server and agents can be one minor version apart.

For example, let’s say you want to upgrade from 2.5.11 to 2.6.x. Start by upgrading your management server to the latest patch version of the 2.6 minor release. Your management server and agent are still compliant as they are one minor version apart. Then, roll out the 2.6 minor release upgrade to the agents in your workload clusters.

If you plan to upgrade more than one minor releases, you must perform one minor release upgrade at a time. For example, to upgrade your management server and agent from 2.3.x to 2.5.x, you upgrade your management server to the latest patch version of the 2.4 minor release first. Your management server and agent are compliant because they are one minor version apart. Then, you upgrade your agents to the 2.4 minor release. After you verify the 2.4 upgrade, use the same approach to upgrade the management server and agents from 2.4 to the target 2.5 minor release.

If both your management server and agent run the same minor version, the agent can run any patch version that is equal or lower than the management server’s patch version.

Consider the following example version skew scenarios:

Supported?Management server versionAgent versionRequirement
✅2.5.92.5.3The management server and agents run the same minor version. The agent patch version is equal to or lower than the management server.
❌2.5.72.5.9The agent runs the same minor version as the server, but has a patch version greater than the server.
✅2.5.92.4.16The agent runs a minor version no greater than n-1 behind the server.
❌2.5.92.3.24The agent runs a minor version that is greater than n-1 behind the server.

Step 1: Prepare to upgrade

  1. Review changes from the previous version.

  2. Check that your underlying Kubernetes platform and Istio service mesh run supported versions for the Gloo Mesh Gateway version that you want to upgrade to.

    1. Review the supported versions.
    2. Compare the supported version against the versions of Kubernetes and Istio that you run in your clusters.
    3. If necessary, upgrade Istio or Kubernetes to a version that is supported by the Gloo Mesh Gateway version that you want to upgrade to.

  3. Set the Gloo Mesh Gateway version that you want to upgrade to as an environment variable. The latest version is used as an example. You can find other versions in the changelog documentation. Append -fips for a FIPS-compliant image, such as 2.6.5-fips. Do not include v before the version number.

      export UPGRADE_VERSION=2.6.5

Step 2: Upgrade the meshctl CLI

Upgrade the meshctl CLI to the version of Gloo Mesh Gateway you want to upgrade to.

  1. Re-install meshctl to the upgrade version.

      curl -sL https://run.solo.io/meshctl/install | GLOO_MESH_VERSION=v$UPGRADE_VERSION sh -

  2. Verify that the client version matches the version you installed.

      meshctl version

    Example output:

      {
"client": {
  "version": "2.6.5"
},

Step 3: Upgrade Gloo Mesh Gateway

Upgrade your Gloo Mesh Gateway installation. The steps differ based on whether you run Gloo Mesh Gateway in a single-cluster or multicluster environment.

Single cluster

  1. Scale down the Gloo management server to 0 replicas. This step is required to safely enable safe mode and I/O threads in Redis. If you already enabled safe mode and you do not want to set I/O threads in Redis, you can skip this step.

      kubectl scale deployment gloo-mesh-mgmt-server --replicas=0 -n gloo-mesh

  2. Update the gloo-platform Helm repo.

      helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts
helm repo update

  3. Apply the Gloo custom resource definitions (CRDs) for the upgrade version.

      helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
    --namespace gloo-mesh \
    --version $UPGRADE_VERSION \
    --set installEnterpriseCrds=true

  4. Get the Helm values file for your current version.

      helm get values gloo-platform -o yaml -n gloo-mesh > gloo-single.yaml
open gloo-single.yaml

  5. Compare your current Helm chart values with the version that you want to upgrade to. You can get a values file for the upgrade version with the helm show values command.

      helm show values gloo-platform/gloo-platform --version $UPGRADE_VERSION > all-values.yaml

  6. Make any changes that you want, such as modifications required for breaking changes or to enable new features, by editing your gloo-single.yaml Helm values file or preparing the --set flags. For example, enabling the Gloo insights engine by including the --set glooInsightsEngine.enabled=true and --set glooAnalyzer.enabled=true settings is recommended to help you improve the security and observability of your environment. If you do not want to use certain settings, comment them out.

  7. Upgrade the Gloo Mesh Gateway Helm installation.

      helm upgrade gloo-platform gloo-platform/gloo-platform \
    --namespace gloo-mesh \
    -f gloo-single.yaml \
    --version $UPGRADE_VERSION

  8. Confirm that Gloo components, such as the gloo-mesh-mgmt-server, run the version that you upgraded to.

      meshctl version

    Example output:

      "server": [
{
  "Namespace": "gloo-mesh",
  "components": [
    {
      "componentName": "gloo-mesh-mgmt-server",
      "images": [
         {
          "name": "gloo-mesh-mgmt-server",
          "domain": "gcr.io",
          "path": "gloo-mesh-mgmt-server",
          "version": "2.6.5"
        }
      ]
    },

  9. If you previously scaled down the Gloo management server, scale it back to the number of replicas that you had before. The following example uses 1 replica.

      kubectl scale deployment gloo-mesh-mgmt-server --replicas=1 -n gloo-mesh

  10. Wait until all Gloo agents are connected to the Gloo management server.

      meshctl check

Multicluster

  1. Scale down the Gloo management server to 0 replicas. This step is required to safely enable safe mode and I/O threads in Redis. If you already enabled safe mode and you do not want to set I/O threads in Redis, you can skip this step.

      kubectl scale deployment gloo-mesh-mgmt-server --context $MGMT_CONTEXT --replicas=0 -n gloo-mesh

  2. Update the gloo-platform Helm repo.

      helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts
helm repo update

  3. Get the Helm values files for your current version.

    1. Get your current values for the management cluster.
        helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $MGMT_CONTEXT > mgmt-plane.yaml
open mgmt-plane.yaml
    2. Get your current values for the workload clusters.
        helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $REMOTE_CONTEXT > data-plane.yaml
open data-plane.yaml
    3. Optional: If you maintain a separate gloo-agent-addons Helm release, get the values for that Helm release too. Note that your add-ons might be installed in a different namespace, such as gloo-mesh-addons.
        helm get values gloo-agent-addons -n gloo-mesh -o yaml --kube-context $REMOTE_CONTEXT > gloo-agent-addons.yaml
open gloo-agent-addons.yaml

  4. Compare your current Helm chart values with the version that you want to upgrade to. You can get a values file for the upgrade version with the helm show values command.

      helm show values gloo-platform/gloo-platform --version $UPGRADE_VERSION > all-values.yaml

  5. Make any changes that you want, such as modifications required for breaking changes or to enable new features, by editing your mgmt-plane.yaml and data-plane.yaml Helm values files or preparing the --set flags. For example, enabling the Gloo insights engine by including the --set glooInsightsEngine.enabled=true setting is recommended to help you improve the security and observability of your environment. If you do not want to use certain settings, comment them out.

  6. Upgrade the Gloo Mesh Gateway Helm release in your management cluster.

    1. Apply the Gloo custom resource definitions (CRDs) for the upgrade version in the management cluster.
        helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
    --kube-context $MGMT_CONTEXT \
    --namespace gloo-mesh \
    --version $UPGRADE_VERSION \
    --set installEnterpriseCrds=true
    2. Upgrade your Helm release in the management cluster. Make sure to include your Helm values when you upgrade either as a configuration file in the --values flag or with --set flags. Otherwise, any previous custom values that you set might be overwritten.
        helm upgrade gloo-platform gloo-platform/gloo-platform \
    --kube-context $MGMT_CONTEXT \
    --namespace gloo-mesh \
    -f mgmt-plane.yaml \
    --version $UPGRADE_VERSION
    3. Confirm that the management plane components, such as the gloo-mesh-mgmt-server, run the version that you upgraded to.
        meshctl version --kubecontext $MGMT_CONTEXT
      Example output:
        "server": [
{
  "Namespace": "gloo-mesh",
  "components": [
    {
      "componentName": "gloo-mesh-mgmt-server",
      "images": [
         {
          "name": "gloo-mesh-mgmt-server",
          "domain": "gcr.io",
          "path": "gloo-mesh-mgmt-server",
          "version": "2.6.5"
        }
      ]
    },

  7. If you previously scaled down the Gloo management server, scale it back to the number of replicas that you had before. The following example uses 1 replica.

      kubectl scale deployment gloo-mesh-mgmt-server --context $MGMT_CONTEXT --replicas=1 -n gloo-mesh

  8. Wait until all Gloo agents are connected to the Gloo management server.

      meshctl check --kubecontext $REMOTE_CONTEXT

  9. Open the Gloo UI and verify that none of your clusters are currently in safe mode. If a cluster is in safe more, the agent’s input snapshot is not yet populated in Redis. You see a yellow banner in the Gloo UI indicating the cluster that triggered safe mode. Wait until the input snapshot is populated and the yellow banner does not display in the Gloo UI anymore.

      meshctl dashboard --kubecontext $MGMT_CONTEXT

  10. Upgrade the Gloo Mesh Gateway Helm releases in your workload clusters. Repeat these steps for each workload cluster, and be sure to update the cluster context each time.

    1. Apply the Gloo custom resource definitions (CRDs) for the upgrade version in each workload cluster.

        helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
   --kube-context $REMOTE_CONTEXT \
   --namespace=gloo-mesh \
   --version=$UPGRADE_VERSION \
   --set installEnterpriseCrds=true

    2. Upgrade your Helm release in each workload cluster. Make sure to include your Helm values when you upgrade either as a configuration file in the --values flag or with --set flags. For example, enabling the Gloo insights analyzer by including the --set glooAnalyzer.enabled=true setting is recommended to help you improve the security and observability of your environment. Otherwise, any previous custom values that you set might be overwritten.

        helm upgrade gloo-platform gloo-platform/gloo-platform \
   --kube-context $REMOTE_CONTEXT \
   --namespace gloo-mesh \
   -f data-plane.yaml \
   --version $UPGRADE_VERSION

    3. Optional: If you maintain a separate gloo-agent-addons Helm release, upgrade that Helm release in each workload cluster too. Be sure to update the cluster context for each workload cluster that you repeat this command for. Note that your add-ons might be installed in a different namespace, such as gloo-mesh-addons.

        helm upgrade gloo-agent-addons gloo-platform/gloo-platform \
  --kube-context $REMOTE_CONTEXT \
  --namespace gloo-mesh \
  -f gloo-agent-addons.yaml \
  --version $UPGRADE_VERSION

    4. Ensure that the Gloo agent connects to the Gloo management server.

        meshctl check --kubecontext $REMOTE_CONTEXT

    5. Open the Gloo UI and verify that none of your clusters are currently in safe mode. If a cluster is in safe more, the agent’s input snapshot is not yet populated in Redis. You see a yellow banner in the Gloo UI indicating the cluster that triggered safe mode. Wait until the input snapshot is populated and the yellow banner does not display in the Gloo UI anymore.

        meshctl dashboard --kubecontext $MGMT_CONTEXT

    6. Confirm that the data plane components, such as the gloo-mesh-agent, run the version that you upgraded to.

        meshctl version --kubecontext $REMOTE_CONTEXT

      Example output:

        {
     "componentName": "gloo-mesh-agent",
     "images": [
       {
         "name": "gloo-mesh-agent",
         "domain": "gcr.io",
         "path": "gloo-mesh/gloo-mesh-agent",
         "version": "2.6.5"
       }
     ]
   },

    7. Repeat these steps for each workload cluster, and be sure to update the cluster context each time.

Update your Gloo license

Before your Gloo license expires, you can update the license by patching the license key secret. If you use Gloo Mesh along with other Gloo products such as Gloo Mesh Gateway, you can also update those licenses.

For example, if you notice that your Gloo management plane deployments are in a crash loop, your Gloo license might be expired. You can check the status of your license with the meshctl license check command.

  • To pass in a license key directly, encode the key in base64 and pass it in the --key flag. For example, to check your Gloo Mesh Enterprise license key, you can run the following command:
      meshctl license check --key $(echo ${GLOO_MESH_LICENSE_KEY} | base64 -w0) --context ${MGMT_CONTEXT}
  • If you store your license keys in a Kubernetes secret, you can pass the secret YAML file in the --secrets-file flag instead.
      meshctl license check --secrets-file license-keys.yaml --context ${MGMT_CONTEXT}

Example output for an expired license:

  WARNING  Your gloo-mesh license expired on 2024-01-24 19:30:53 +0100 CET. To get a new license, contact Support.
ERROR  License is expired. For more info, see https://docs.solo.io/gloo-mesh-enterprise/latest/setup/prepare/licensing/#update-licenses

To update your license key in your Gloo installation:

  1. Get a new Gloo license key by contacting your account representative. If you use Gloo Mesh along with other Gloo products such as Gloo Mesh Gateway, make sure to ask for up-to-date license keys for all your products.

  2. Save the new license key as an environment variable.

      export GLOO_MESH_CORE_LICENSE_KEY=<new-key-string>
  
      export GLOO_MESH_LICENSE_KEY=<new-key-string>
  
      export GLOO_MESH_GATEWAY_LICENSE_KEY=<new-key-string>

  3. Update the license secret to use the new Gloo license key.

      kubectl -n gloo-mesh patch secret license-keys -p "stringData: { gloo-mesh-core-license-key: $GLOO_MESH_CORE_LICENSE_KEY }"
  
      kubectl -n gloo-mesh patch secret license-keys -p "stringData: { gloo-mesh-license-key: $GLOO_MESH_LICENSE_KEY }"
  
      kubectl -n gloo-mesh patch secret license-keys -p "stringData: { gloo-gateway-license-key: $GLOO_MESH_GATEWAY_LICENSE_KEY }"

  4. Optional: If your license expired and the management server pods are in a crash loop, restart the management server pods. If you updated the license before expiration, skip this step.

      kubectl rollout restart -n gloo-mesh deployment/gloo-mesh-mgmt-server --context ${MGMT_CONTEXT}

  5. Verify that your license check is now valid, and no errors are reported.

    • To pass in a license key directly, encode the key in base64 and pass it in the --key flag. For example, to check your Gloo Mesh Enterprise license key, you can run the following command:
        meshctl license check --key $(echo ${GLOO_MESH_LICENSE_KEY} | base64 -w0) --context ${MGMT_CONTEXT}
    • If you store your license keys in a Kubernetes secret, you can pass the secret YAML file in the --secrets-file flag instead.
        meshctl license check --secrets-file license-keys.yaml --context ${MGMT_CONTEXT}

    Example output:

      INFO  License key gloo-mesh-license-key for product gloo-mesh is valid. Expires at 08 Oct 24 12:31 CEST
SUCCESS  Licenses are valid

Upgrade managed gateways within your gloo-platform Helm chart

If you manage Istio installations within the istioInstallations section your Gloo Platform Helm chart, you can apply updates to your Istio installations in one of the following ways:

In-place upgrades

In an in-place upgrade, Gloo upgrades your existing control plane or gateway installations. Note that in production environments, canary upgrades are recommended for updating the minor version. However, you might want to use in-place upgrades for patch versions or changes within the same minor version. Be sure to test in-place upgrades in development or staging environments first.

In-place upgrades behave differently based on whether your installation is revisionless or revisioned.

Revisionless installations (testing only): If your testing-only installation is revisionless (your settings omit the revision and gatewayRevision fields), in-place upgrades are triggered when you apply changes to any field in the istioInstallations section.

Revisioned installations: If your installation is revisioned, in-place upgrades are triggered only when you apply changes to one of the following fields in the istioInstallations section. Otherwise, a canary upgrade is required.You can change this behavior by setting the skipUpgradeValidation boolean value to true for the Istio installation.

  • istioOperatorSpec.tag patch version
    • In-place upgrades are not supported for downgrading the patch version.
    • In-place upgrades are not supported if you do not already specify a tag value, such as if you want to switch from the auto setting to a specific version. This is because you must also specify hub and revision values, which require a canary upgrade.
  • istioOperatorSpec.meshConfig values

To trigger an in-place upgrade:

  1. Follow the steps in this guide to perform a regular upgrade of your Gloo Mesh installation and include your Istio changes in your Helm values file. For example, in a single-cluster setup, you might edit your Helm values file to update the patch version of Istio in the istioInstallations.controlPlane.installations.istioOperatorSpec.tag and istioInstallations.northSouthGateways.installations.istioOperatorSpec.tag fields. After you apply the updates in your Helm upgrade of the gloo-platform chart, Gloo starts an in-place upgrade of the Istio control plane and gateways.

  2. Verify that your Istio resources are updated.

      kubectl get all -n gm-iop-1-22 --context $REMOTE_CONTEXT1
kubectl get all -n istio-system --context $REMOTE_CONTEXT1
kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1

Revisioned canary upgrades

In a canary upgrade, you install another Istio installation (canary) alongside your active installation. Each installation is revisioned so that you can easily identify and verify the separate settings and resources for each installation. Note that during a canary upgrade, the validating admissions webhook is enabled only for the canary installation to prevent issues that occur when multiple webhooks are enabled.

Perform a canary upgrade when you change one of the following fields:

  • istioOperatorSpec.tag minor version
  • istioOperatorSpec.hub repository, such as switching to the repository for the minor version of the Solo distribution of Istio that you want to upgrade to
  • components, profile, values, or namespace in the istioOperatorSpec

To perform a canary upgrade:

  1. OpenShift only: Elevate the permissions of the service account that will be created for the new revision’s operator project. This permission allows the Istio sidecar to make use of a user ID that is normally restricted by OpenShift. Replace the revision with the revision you plan to use.

      oc adm policy add-scc-to-group anyuid system:serviceaccounts:gm-iop-1-22 --context $REMOTE_CONTEXT1
oc adm policy add-scc-to-group anyuid system:serviceaccounts:gm-iop-1-22 --context $REMOTE_CONTEXT2

  2. Follow the steps in this guide to perform a regular upgrade of your Gloo Mesh installation. When you edit the istioInstallations.controlPlane and istioInstallations.eastWestGateways sections of your Helm values file, add another installation entry for the canary revision, and leave the entry your current installation as-is. For the canary revision, be sure to set defaultRevision and activeGateway to false so that only the existing revisions continue to run.

    For example, you might add the following installation entries for the Istio control plane and east-west gateway alongside your existing entries. If you have a Gloo Mesh Gateway license, you might also have entries for the ingress gateway proxy in the nothSouthGateways section too.

      
istioInstallations:
  controlPlane:
    enabled: true
    installations:
      # EXISTING revision
      - clusters:
          - defaultRevision: true # Keep this field set to TRUE
            name: cluster1
            trustDomain: ""
        istioOperatorSpec:
          hub: $REPO
          tag: 1.21.5-solo
          profile: minimal
          namespace: istio-system
          ...
        revision: 1-21
      # NEW revision
      - clusters:
          - defaultRevision: false # Set this field to FALSE
            name: cluster1
            trustDomain: ""
        istioOperatorSpec:
          hub: $REPO
          tag: 1.22.4-solo
          profile: minimal
          namespace: istio-system
          ...
        revision: 1-22
  eastWestGateways:
    - enabled: true
      installations:
        # EXISTING revision
        - clusters:
            - activeGateway: true # Keep this field set to TRUE
              name: cluster1
              name: 
              trustDomain: ""
          gatewayRevision: 1-21
          istioOperatorSpec:
            hub: $REPO
            tag: 1.21.5-solo
            profile: empty
            namespace: gloo-mesh-gateways
            ...
        # NEW revision
        - clusters:
            - defaultRevision: false # Set this field to FALSE
              name: cluster1
              name: 
              trustDomain: ""
          gatewayRevision: 1-22
          istioOperatorSpec:
            hub: $REPO
            tag: 1.22.4-solo
            profile: empty
            namespace: gloo-mesh-gateways
            ...
      name: istio-eastwestgateway
  enabled: true
    • Updating the minor version of Istio? In your canary revision section, be sure to update both the repo key in the hub field, and the Istio version in the tag field. You can get the repo key for the Istio version that you want to install from the Istio images built by Solo.io support article.
    • For most use cases, you can set the revision and the gatewayRevision to the same version. However, gateway installations can point to any istiod control plane revision by using the controlPlaneRevision field. For simplicity, if you do not specify controlPlaneRevision, the gateway installation uses a control plane with the same revision as itself.

  3. After you apply the Helm upgrade with your updated values file, verify that Istio resources for the canary installation are created. For example, if you updated the Istio minor version to 1-22, verify that resources are created in the gm-iop-1-22 namespace, and that resources for 1-22 are created alongside the existing resources for the previous version in the istio-system and gloo-mesh-gateways namespaces. Note that the gateway load balancers for the canary revision contain the revision in the name, such as istio-eastwestgateway-1-22.

      kubectl get all -n gm-iop-1-22 --context $REMOTE_CONTEXT
kubectl get all -n istio-system --context $REMOTE_CONTEXT
kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT

  4. After performing any necessary testing, switch to the new Istio control plane and gateway revisions.

    1. Get your Helm values file. Change the release name as needed.
        helm get values gloo-platform -n gloo-mesh -o yaml > mgmt-plane.yaml
open mgmt-plane.yaml
    2. Change defaultRevision and activeGateway to false for the old revision and to true for the new revision.
      • New load balancers are created for the canary gateways. To instead change the control plane revision in use by the existing gateway load balancers, you can set the istio.io/rev label on the gateway deployment, which triggers a rolling restart.
        
istioInstallations:
  controlPlane:
    enabled: true
    installations:
      # EXISTING revision
      - clusters:
          - defaultRevision: false # Set this field to FALSE
            name: cluster1
            trustDomain: ""
        istioOperatorSpec:
          hub: $REPO
          tag: 1.21.5-solo
          profile: minimal
          namespace: istio-system
          ...
        revision: 1-21
      # NEW revision
      - clusters:
          - defaultRevision: true # Set this field to TRUE
            name: cluster1
            trustDomain: ""
        istioOperatorSpec:
          hub: $REPO
          tag: 1.22.4-solo
          profile: minimal
          namespace: istio-system
          ...
        revision: 1-22
  eastWestGateways:
    - enabled: true
      installations:
        # EXISTING revision
        - clusters:
            - activeGateway: false # Set this field set to FALSE
              name: cluster1
              name: 
              trustDomain: ""
          gatewayRevision: 1-21
          istioOperatorSpec:
            hub: $REPO
            tag: 1.21.5-solo
            profile: empty
            namespace: gloo-mesh-gateways
            ...
        # NEW revision
        - clusters:
            - defaultRevision: true # Set this field to TRUE
              name: cluster1
              name: 
              trustDomain: ""
          gatewayRevision: 1-22
          istioOperatorSpec:
            hub: $REPO
            tag: 1.22.4-solo
            profile: empty
            namespace: gloo-mesh-gateways
            ...
      name: istio-eastwestgateway
  enabled: true
    3. Upgrade your Helm release. Change the release name as needed.
        helm upgrade gloo-platform gloo-platform/gloo-platform \
   --namespace gloo-mesh \
   -f mgmt-plane.yaml \
   --version $UPGRADE_VERSION

  5. After your Helm upgrade completes, verify that the active gateways for the new revision are created, which do not have the revision appended to the name. Note that gateways for the inactive revision that you previously ran also exist in the namespace, in the case that a rollback is required.

      kubectl get all -n gloo-mesh-gateways

    Example output, in which the active gateway (istio-eastwestgateway) for the new revision and inactive gateway (such as istio-eastwestgateway-1-21) for the old revision are created:

      NAME                            TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)     
istio-eastwestgateway            LoadBalancer   10.44.4.140   34.150.235.221   15021:31321/TCP,80:32525/TCP,443:31826/TCP   48s                                                 AGE
istio-eastwestgateway-1-21       LoadBalancer   10.56.15.36    34.145.163.61   15021:31936/TCP,80:30196/TCP,443:32286/TCP,15443:31851/TCP   45s

  6. Upgrade your apps’ Istio sidecars.

    1. For any namespace where you run Istio-managed apps, change the label to use the new revision. For example, you might update the bookinfo namespace for the canary revision 1-22. If you did not previously use revision labels for your apps, you can upgrade your application’s sidecars by running kubectl label ns bookinfo istio-injection- and kubectl label ns bookinfo istio.io/rev=<revision>.
      • Single cluster setup:
          kubectl label ns bookinfo --overwrite istio.io/rev=1-22
      • Multicluster setup:
          kubectl label ns bookinfo --overwrite istio.io/rev=1-22 --context $REMOTE_CONTEXT1
kubectl label ns bookinfo --overwrite istio.io/rev=1-22 --context $REMOTE_CONTEXT2
    2. Update any Istio-managed apps by rolling out restarts. The Istio sidecars for each microservice are updated to use the new Istio version. Make sure that you only restart one microservice at a time. For example, in the following commands to update the Bookinfo microservices, 20 seconds elapse between each restart to ensure that the pods have time to start running.
      • Single cluster setup:
          kubectl rollout restart deployment -n bookinfo details-v1
sleep 20s
kubectl rollout restart deployment -n bookinfo productpage-v1
sleep 20s
kubectl rollout restart deployment -n bookinfo reviews-v1
sleep 20s
kubectl rollout restart deployment -n bookinfo reviews-v2
sleep 20s
kubectl rollout restart deployment -n bookinfo reviews-v3
sleep 20s
kubectl rollout restart deployment -n bookinfo ratings-v1
sleep 20s
      • Multicluster setup:
          kubectl rollout restart deployment -n bookinfo details-v1 --context $REMOTE_CONTEXT1
sleep 20s
kubectl rollout restart deployment -n bookinfo ratings-v1 --context $REMOTE_CONTEXT1
sleep 20s
kubectl rollout restart deployment -n bookinfo productpage-v1 --context $REMOTE_CONTEXT1
sleep 20s
kubectl rollout restart deployment -n bookinfo reviews-v1 --context $REMOTE_CONTEXT1
sleep 20s
kubectl rollout restart deployment -n bookinfo reviews-v2 --context $REMOTE_CONTEXT1
sleep 20s
kubectl rollout restart deployment -n bookinfo reviews-v3 --context $REMOTE_CONTEXT2
sleep 20s
kubectl rollout restart deployment -n bookinfo ratings-v1 --context $REMOTE_CONTEXT2
sleep 20s
    3. Verify that your workloads and new gateways point to the new revision.
      • Single cluster setup:
          istioctl proxy-status
      • Multicluster setup:
          istioctl proxy-status --context $REMOTE_CONTEXT1
      Example output:
        NAME                                                              CLUSTER     ...     ISTIOD                         VERSION
details-v1-7b6df9d8c8-s6kg5.bookinfo                              cluster1    ...     istiod-1-22-7c8f6fd4c4-m9k9t     1.22.4-solo
istio-eastwestgateway-1-22-bdc4fd65f-ftmz9.gloo-mesh-gateways     cluster1    ...     istiod-1-22-6495985689-rkwwd     1.22.4-solo
productpage-v1-bb494b7d7-xbtxr.bookinfo                           cluster1    ...     istiod-1-22-7c8f6fd4c4-m9k9t     1.22.4-solo
ratings-v1-55b478cfb6-wv2m5.bookinfo                              cluster1    ...     istiod-1-22-7c8f6fd4c4-m9k9t     1.22.4-solo
reviews-v1-6dfcc9fc7d-7k6qh.bookinfo                              cluster1    ...     istiod-1-22-7c8f6fd4c4-m9k9t     1.22.4-solo
reviews-v2-7dddd799b5-m5n2z.bookinfo                              cluster1    ...     istiod-1-22-7c8f6fd4c4-m9k9t     1.22.4-solo

  7. To uninstall the previous installations, or if you need to uninstall the canary installations, you can edit your Helm values file to remove the revision entries from the istioInstallations.controlPlane.installations and istioInstallations.northSouthGateways.installations lists. Then, upgrade your Gloo Mesh Helm release with your updated values file.

Testing only: Manually replacing the lifecycle manager CRs

If you manage Istio through your main Gloo Platform Helm chart in testing or demo setups, you can quickly upgrade your Istio service mesh and gateway configurations by manually deleting the IstioLifecycleManager and GatewayLifecycleManager CRs, and upgrading your Gloo Mesh installation with your updated gateway values in your Helm values file. Note that you can also use this method to clear your managed Istio configurations if a canary upgrade becomes stuck.

  1. Get the name of your IstioLifecycleManager resource. Typically, this resource is named gloo-platform.

      kubectl get IstioLifecycleManager -A --context $MGMT_CONTEXT

  2. Delete the resource.

      kubectl delete IstioLifecycleManager gloo-platform -n gloo-mesh --context $MGMT_CONTEXT

  3. Verify that your istiod control plane is removed.

      kubectl get all -n istio-system --context $REMOTE_CONTEXT1
kubectl get all -n istio-system --context $REMOTE_CONTEXT2

  4. Optional: If you also need to make changes to your gateways, clear those configurations.

    1. Get the name of your GatewayLifecycleManager resource. Typically, this resource is named istio-eastwestgateway. You might also have an istio-ingressgateway resource, such as if you use Gloo Mesh Gateway.
        kubectl get GatewayLifecycleManager -A --context $MGMT_CONTEXT
    2. Delete the resource.
        kubectl delete GatewayLifecycleManager istio-eastwestgateway -n gloo-mesh --context $MGMT_CONTEXT
  
        kubectl delete GatewayLifecycleManager istio-ingressgateway -n gloo-mesh --context $MGMT_CONTEXT
    3. Verify that your gateway proxy is removed.
        kubectl get svc -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
kubectl get svc -n gloo-mesh-gateways --context $REMOTE_CONTEXT2

  5. Follow the steps in this guide to perform a regular upgrade of your Gloo Mesh installation and include your Istio changes in your Helm values file. After you apply the updates in your Helm upgrade of the gloo-platform chart, Gloo re-creates the istiod control plane, and if applicable, the Istio gateways.

  6. After your Helm upgrade completes, verify that your Istio resources are re-created.

      # Change the revision as needed
kubectl get all -n gm-iop-1-22 --context $REMOTE_CONTEXT1
kubectl get all -n istio-system --context $REMOTE_CONTEXT1
kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
  
      # Change the revision as needed
kubectl get all -n gm-iop-1-22 --context $REMOTE_CONTEXT2
kubectl get all -n istio-system --context $REMOTE_CONTEXT2
kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT2

Upgrade gateways with lifecycle manager CRs

If you manually created IstioLifecycleManager and GatewayLifecycleManager CRs to deploy gateway proxies, you can apply updates to your gateway proxies by using an in-place upgrade or a revisioned canary upgrade.

In-place upgrades

In an in-place upgrade, Gloo upgrades your existing control plane or gateway installations. In-place upgrades behave differently based on whether your installation is revisionless or revisioned.

Revisionless (testing only): If your testing-only installation is revisionless (your settings omit the revision and gatewayRevision fields), in-place upgrades are triggered when you apply changes to any field in the istioInstallations section.

Revisioned: If your installation is revisioned, in-place upgrades are triggered only when you apply changes to one of the following fields in the istioInstallations section. Otherwise, a canary upgrade is required. You can change this behavior by setting the spec.installations.skipUpgradeValidation boolean value to true in the IstioLifecycleManager and GatewayLifecycleManager resources.

  • istioOperatorSpec.tag patch version
    • In-place upgrades are not supported for downgrading the patch version.
    • In-place upgrades are not supported if you do not already specify a tag value, such as if you want to switch from the auto setting to a specific version. This is because you must also specify hub and revision values, which require a canary upgrade.
  • istioOperatorSpec.meshConfig values

  1. Get the names of your managed Istio installation resources.

      kubectl get IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT
kubectl get GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT

  2. Upgrade your istiod control plane by editing the IstioLifecycleManager resource. For example, you might update the patch version of Istio by changing the value of istioOperatorSpec.tag. After you save and close the editor, Gloo starts an in-place upgrade of the istiod control planes.

      kubectl edit IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <istiod-installation-name>

  3. If you created GatewayLifecycleManager resources for ingress, egress, or east-west gateways, you can upgrade your gateways by editing each resource. For example, if you updated the patch version of the control plane, you can also update your gateway patch versions by changing the value of istioOperatorSpec.tag. After you save and close the editor, Gloo starts an in-place upgrade of the gateways.

      kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <ingress-gateway-installation-name>
  
      kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <egress-gateway-installation-name>

  4. Verify that your Istio resources are updated.

      kubectl get all -n gm-iop-1-22 --context $REMOTE_CONTEXT1
kubectl get all -n istio-system --context $REMOTE_CONTEXT1
kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1

Revisioned canary upgrades

In a canary upgrade, you install another Istio installation (canary) alongside your active installation. Each installation is revisioned so that you can easily identify and verify the separate settings and resources for each installation.

Perform a canary upgrade when you change one of the following fields:

  • istioOperatorSpec.tag minor version
  • istioOperatorSpec.hub repository, such as switching to the repository for the minor version of the Solo distribution of Istio that you want to upgrade to
  • components, profile, values, or namespace in the istioOperatorSpec

Keep in mind the following considerations:

  • During a canary upgrade, the validating admissions webhook is enabled only for the canary installation to prevent issues that occur when multiple webhooks are enabled.
  • If you plan to run multiple revisions of Istio in your cluster and use discoverySelectors in each revision to discover the resources in specific namespaces, enable the glooMgmtServer.extraEnvs.IGNORE_REVISIONS_FOR_VIRTUAL_DESTINATION_TRANSLATION environment variable on the Gloo management server. For more information, see Multiple Istio revisions in the same cluster.

  1. OpenShift only: Elevate the permissions of the service account that will be created for the new revision’s operator project. This permission allows the Istio sidecar to make use of a user ID that is normally restricted by OpenShift. Replace the revision with the revision you plan to use.

      oc adm policy add-scc-to-group anyuid system:serviceaccounts:gm-iop-1-22 --context $REMOTE_CONTEXT1
oc adm policy add-scc-to-group anyuid system:serviceaccounts:gm-iop-1-22 --context $REMOTE_CONTEXT2

  2. Get the names of your managed Istio installation resources.

      kubectl get IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT
kubectl get GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT

  3. Edit the IstioLifecycleManager resource for the istiod control plane to add another installation entry for the canary revision. For the canary revision, be sure to set defaultRevision to false so that only the existing revision continues to run.

      kubectl edit IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <installation-name>

    For example, if you upgrade to Istio 1.22.4, add the following 1-22 canary revision as a new entry in the installations section:

      apiVersion: admin.gloo.solo.io/v2
kind: IstioLifecycleManager
metadata:
  name: istiod-control-plane
  namespace: gloo-mesh
spec:
  installations:
    # Existing revision
    - revision: 1-21
      clusters:
      - name: cluster1
        # Keep this field set to TRUE so that only the existing revision continues to run
        defaultRevision: true
      - name: cluster2
        defaultRevision: true
      istioOperatorSpec:
        ...
    # Canary revision
    - revision: 1-22
      clusters:
      - name: cluster1
        # Set this field to FALSE so that only the existing revision continues to run
        defaultRevision: false
      - name: cluster2
        defaultRevision: false
      istioOperatorSpec:
        ...

  4. If you created GatewayLifecycleManager resources for ingress, egress, or east-west gateways, edit those resources to add another installation entry for the canary revision. Be sure to set activeGateway to false so that only the existing revision continues to run.

      kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <installation-name>

    For example, if you upgrade to Istio 1.22.4, add the following 1-22 canary revision as a new entry in the installations section:

      apiVersion: admin.gloo.solo.io/v2
kind: GatewayLifecycleManager
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  installations:
    # Existing revision
    - gatewayRevision: 1-21
      clusters:
      - name: cluster1
        # Keep this field set to TRUE so that only the existing revision continues to run
        activeGateway: true
      - name: cluster2
        activeGateway: true
      istioOperatorSpec:
        profile: empty
        ...
    # Canary revision
    - gatewayRevision: 1-22
      clusters:
      - name: cluster1
        # Set this field to FALSE so that only the existing revision continues to run
        activeGateway: false
      - name: cluster2
        activeGateway: false
      istioOperatorSpec:
        profile: empty
        ...

  5. Verify that Istio resources for the canary installation are created. For example, if you updated the Istio minor version to 1.22.4, verify that resources are created in the gm-iop-1-22 namespace, and that resources for 1-22 are created alongside the existing resources for the previous version in the istio-system and gloo-mesh-gateways namespaces. Note that the gateway load balancers for the canary revision contain the revision in the name, such as istio-ingressgateway-1-22.

      kubectl get all -n gm-iop-1-22 --context $REMOTE_CONTEXT1
kubectl get all -n istio-system --context $REMOTE_CONTEXT1
kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
  
      kubectl get all -n gm-iop-1-22 --context $REMOTE_CONTEXT2
kubectl get all -n istio-system --context $REMOTE_CONTEXT2
kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT2

  6. After performing any necessary testing, switch to the new istiod control plane revision by changing defaultRevision to false for the old revision and to true for the new revision.

      kubectl edit IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT istiod-control-plane

    Example:

      apiVersion: admin.gloo.solo.io/v2
kind: IstioLifecycleManager
metadata:
  name: istiod-control-plane
  namespace: gloo-mesh
spec:
  installations:
    # Old revision
    - revision: 1-21
      clusters:
      - name: cluster1
        # Set this field to FALSE
        defaultRevision: false
      - name: cluster2
        defaultRevision: false
      istioOperatorSpec:
        ...
    # New revision
    - revision: 1-22
      clusters:
      - name: cluster1
        # Set this field to TRUE
        defaultRevision: true
      - name: cluster2
        defaultRevision: true
      istioOperatorSpec:
        ...

  7. In your workload clusters, update the labels on your service namespaces to use the new revision.

    1. For any namespace where you run Istio-managed apps, change the label to use the new revision, such as in these example commands.

        kubectl label ns <namespace> --overwrite istio.io/rev=1-22 --context $REMOTE_CONTEXT1
kubectl label ns <namespace> --overwrite istio.io/rev=1-22 --context $REMOTE_CONTEXT2

      For example, if you installed the Bookinfo sample app:

        kubectl label ns bookinfo --overwrite istio.io/rev=1-22 --context $REMOTE_CONTEXT1
kubectl label ns bookinfo --overwrite istio.io/rev=1-22 --context $REMOTE_CONTEXT2

    2. Update any Istio-managed apps by rolling out restarts. The Istio sidecars for each microservice are updated to use the new Istio version. Make sure that you only restart one microservice at a time. For example, in the following commands to update the Bookinfo microservices, 20 seconds elapse between each restart to ensure that the pods have time to start running.

        kubectl rollout restart deployment -n bookinfo details-v1 --context $REMOTE_CONTEXT1
sleep 20s
kubectl rollout restart deployment -n bookinfo ratings-v1 --context $REMOTE_CONTEXT1
sleep 20s
kubectl rollout restart deployment -n bookinfo productpage-v1 --context $REMOTE_CONTEXT1
sleep 20s
kubectl rollout restart deployment -n bookinfo reviews-v1 --context $REMOTE_CONTEXT1
sleep 20s
kubectl rollout restart deployment -n bookinfo reviews-v2 --context $REMOTE_CONTEXT1
sleep 20s
kubectl rollout restart deployment -n bookinfo reviews-v3 --context $REMOTE_CONTEXT2
sleep 20s
kubectl rollout restart deployment -n bookinfo ratings-v1 --context $REMOTE_CONTEXT2
sleep 20s

    3. Verify that your workloads and new gateways point to the new revision.

        istioctl proxy-status --context $REMOTE_CONTEXT1

      Example output:

        NAME                                                              CLUSTER     ...     ISTIOD                           VERSION
details-v1-7b6df9d8c8-s6kg5.bookinfo                              cluster1    ...     istiod-1-22-7c8f6fd4c4-m9k9t     1.22.4-solo
istio-ingressgateway-1-22-bdc4fd65f-ftmz9.gloo-mesh-gateways    cluster1    ...     istiod-1-22-6495985689-rkwwd     1.22.4-solo
productpage-v1-bb494b7d7-xbtxr.bookinfo                           cluster1    ...     istiod-1-22-7c8f6fd4c4-m9k9t     1.22.4-solo
ratings-v1-55b478cfb6-wv2m5.bookinfo                              cluster1    ...     istiod-1-22-7c8f6fd4c4-m9k9t     1.22.4-solo
reviews-v1-6dfcc9fc7d-7k6qh.bookinfo                              cluster1    ...     istiod-1-22-7c8f6fd4c4-m9k9t     1.22.4-solo
reviews-v2-7dddd799b5-m5n2z.bookinfo                              cluster1    ...     istiod-1-22-7c8f6fd4c4-m9k9t     1.22.4-solo

  8. Switch to any new gateways by changing activeGateway to false for the old revision and to true for the new revision.

      kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <installation-name>

    Example:

      apiVersion: admin.gloo.solo.io/v2
kind: GatewayLifecycleManager
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  installations:
    # Old revision
    - gatewayRevision: 1-21
      clusters:
      - name: cluster1
        # Set this field set to FALSE
        activeGateway: false
      - name: cluster2
        activeGateway: false
      istioOperatorSpec:
        profile: empty
        ...
    # New revision
    - gatewayRevision: 1-22
      clusters:
      - name: cluster1
        # Set this field to TRUE
        activeGateway: true
      - name: cluster2
        activeGateway: true
      istioOperatorSpec:
        profile: empty
        ...

  9. Verify that the active gateways for the new revision are created, which do not have the revision appended to the name. Note that gateways for the inactive revision that you previously ran also exist in the namespace, in the case that a rollback is required.

      kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT2

    Example output, in which the active gateway (istio-ingressgateway) for the new revision and inactive gateway (such as istio-ingressgateway-1-21) for the old revision are created:

      NAME                            TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                                                      AGE
istio-ingressgateway            LoadBalancer   10.44.4.140    34.150.235.221  15021:31321/TCP,80:32525/TCP,443:31826/TCP                   48s
istio-ingressgateway-1-21       LoadBalancer   10.56.15.36    34.145.163.61   15021:31936/TCP,80:30196/TCP,443:32286/TCP,15443:31851/TCP   45s

  10. To uninstall the previous installation, or if you need to uninstall the canary installation, you can edit the IstioLifecycleManager and GatewayLifecycleManager resources to remove the revision’s entry from the installations list.