Upgrade Gloo Mesh

Upgrade Gloo Platform minor and patch versions for the management server and agent in each cluster.

Your clusters must run Gloo Mesh Enterprise version 2.1.4 before you can upgrade to version 2.2.x.

You must always upgrade the Gloo management server before upgrading the Gloo agent to avoid unexpected behavior. Note that only n-1 minor version skew is supported between the management server and the agent. For more information, see the Skew policy.

During the upgrade, the data plane continues to run, but you might not be able to modify the configurations through the management plane. Because zero downtime is not guaranteed, try testing the upgrade in a staging environment before upgrading your production environment.

Looking to update certain Helm chart values but not the version? See Modifying Helm chart values.

Before you begin

  1. Review the changelog. Focus especially on any Breaking Changes that might require a different upgrade procedure.

    If you are on version 1.0 or earlier, uninstall and re-install Gloo Mesh Enterprise, as described in one of the setup guides.

  2. Check that your underlying Kubernetes platform and Istio service mesh run supported versions for the Gloo Mesh Enterprise version that you want to upgrade to.

    1. Review the supported versions.
    2. Compare the supported version against the versions of Kubernetes and Istio that you run in your clusters.
    3. If necessary, upgrade Istio or Kubernetes. For example, if you want to upgrade from Gloo Mesh Enterprise 2.0 to 2.1 but you currently run Istio 1.8, you must upgrade to Istio 1.9 first.
      • For Istio steps, see Upgrade Istio.
      • For Kubernetes steps, consult your cluster infrastructure provider.
  3. In your terminal, set environment variables for your current and target Gloo Mesh Enterprise versions to use in the subsequent steps.

    # Set the Gloo Mesh Enterprise version. The latest version is used as an example. You can find other versions in the Changelog documentation.
    export UPGRADE_VERSION=2.2.9
    
    # Save the kubeconfig contexts for your clusters. Run kubectl config get-contexts, look for your cluster in the CLUSTER column, and get the context name in the NAME column. Note: Do not use context names with underscores. The context name is used as a SAN specification in the generated certificate that connects workload clusters to the management cluster, and underscores in SAN are not FQDN compliant. You can rename a context by running kubectl config rename-context "<oldcontext>" <newcontext>.
    export MGMT_CONTEXT=<management_cluster_config>
    export REMOTE_CONTEXT=<remote_cluster_config>
    
    # Set the gloo-mesh namespace.
    export NAMESPACE=gloo-mesh
    
    # Add your Gloo Mesh Enterprise license that you got from your Solo account representative.
    export GLOO_MESH_LICENSE_KEY=<your-key>
    

Step 1: Upgrade the CRDs for your management cluster

  1. Update the Helm repo in the management cluster for Gloo Mesh Enterprise.
    helm repo add gloo-mesh-enterprise https://storage.googleapis.com/gloo-mesh-enterprise/gloo-mesh-enterprise
    helm repo update --kube-context $MGMT_CONTEXT
    
  2. Pull the latest Gloo Mesh Enterprise Helm chart files.
    helm pull gloo-mesh-enterprise/gloo-mesh-enterprise --kube-context $MGMT_CONTEXT --version $UPGRADE_VERSION --untar
    
  3. Apply the Gloo Mesh Enterprise custom resource definitions (CRDs) on your management cluster.
    kubectl apply --context $MGMT_CONTEXT -f gloo-mesh-enterprise/charts/gloo-mesh-crds/crds/
    

If you upgrade to a release such as 2.3.22, you might see an error such as "error":"no matches for kind \"CertificateRequest\" in version \"certificates.mesh.gloo.solo.io/v1\"". The management cluster also needs CRDs for the Gloo Mesh agent. Continue to Step 2, and apply the agent CRDs on your management cluster as well as your remote clusters.

Step 2: Upgrade the CRDs for each workload cluster

  1. Update the Helm repo in your workload cluster for Gloo Mesh Enterprise.
    helm repo add gloo-mesh-agent https://storage.googleapis.com/gloo-mesh-enterprise/gloo-mesh-agent
    helm repo update --kube-context $REMOTE_CONTEXT
    
  2. Pull the latest Gloo Mesh Enterprise agent Helm chart files.
    helm pull gloo-mesh-agent/gloo-mesh-agent --version $UPGRADE_VERSION --kube-context $REMOTE_CONTEXT --untar
    
  3. Apply the Gloo Mesh CRDs on each remote cluster.
    kubectl apply --context $REMOTE_CONTEXT -f gloo-mesh-agent/charts/gloo-mesh-crds/crds
    kubectl apply --context $REMOTE_CONTEXT -f gloo-mesh-agent/charts/ext-auth-service/crds
    kubectl apply --context $REMOTE_CONTEXT -f gloo-mesh-agent/charts/rate-limiter/crds
    
  4. Repeat the previous step for each workload cluster.

Step 3: Get your Helm chart values

As part of the upgrade, you can update or reuse the Helm chart values for your Gloo management server, agent, and any add-ons that you might have such as rate limiting and external authentication services.

Make sure to include your Helm values when you upgrade either as a configuration file in the –values flag or with –set flags. Otherwise, any previous custom values that you set might be overwritten. In single cluster setups, this might mean that your Gloo agent and ingress gateways are removed.
  1. Get your Helm chart values configuration file for your current version. Review the following options:

    • Reuse the existing Helm chart values configuration file for your current installation
    • Get the Helm chart values for your current installation
    • Show the default Helm chart values for the current version

    When you installed Gloo, you might have used a Helm chart values configuration file to customize your installation. This file might be stored in your CI/CD pipeline, such as in a Git repository. The installation instructions used the following file names.

    Helm installations:

    • Management server: values-mgmt-plane-env.yaml, based on this sample values file.
    • Agent: values-data-plane-env.yaml, based on this sample values file.
    • Rate limiting and external authentication add-ons: values.yaml.

    Getting started or meshctl installations:

    • Single cluster: Both Gloo management server and agent in gs-single-values.yaml, based on this sample values file.
    • Multicluster: gs-multi-values.yaml, based on this sample values file. Note that this sample values file was used only for the agent installation, because default Helm values were used for the management server installation. To get a values file for the management server installation, you can run helm -n gloo-mesh get all gloo-mgmt --kube-context $MGMT_CONTEXT > values-mgmt-plane-env.yaml, and use the values listed after COMPUTED VALUES: for your Helm values file.

    If you do not have the original values files that you used at installation time, you can get the values for the current Helm release in your clusters.

    1. Get the values for the current Helm releases.
      • Management server: Note that if the release name gloo-mgmt is not found, your Helm release might be named gloo-mesh-enterprise instead.
        helm -n gloo-mesh get all gloo-mgmt --kube-context $MGMT_CONTEXT > values-mgmt-plane-env.yaml
        open values-mgmt-plane-env.yaml
        
      • Agent: Note that if the release name gloo-agent is not found, your Helm release might be named gloo-mesh-agent instead.
        helm -n gloo-mesh get all gloo-agent --kube-context $REMOTE_CONTEXT > values-data-plane-env.yaml
        open values-data-plane-env.yaml
        
      • Rate limiting and external authentication add-ons:
        helm -n gloo-mesh get all gloo-agent-addons --kube-context $REMOTE_CONTEXT > values.yaml
        open values.yaml
        
    2. In each file, delete the lines up to and including the USER-SUPPLIED VALUES: line, and delete all lines including and after the COMPUTED VALUES line. Save the file.

    You can get the default values for the Helm chart at the version of your installation, which you can check by running meshctl version.

    • Management server:
      helm show values gloo-mesh-enterprise/gloo-mesh-enterprise --version $CURRENT_VERSION > values-mgmt-plane.yaml
      
    • Agent:
      helm show values gloo-mesh-agent/gloo-mesh-agent --version $CURRENT_VERSION > values-data-plane.yaml
      
  2. Compare your current Helm chart values with the version that you want to upgrade to. You can get a values file for the upgrade version with the helm show values command.

    • Management server:
      helm show values gloo-mesh-enterprise/gloo-mesh-enterprise --version $UPGRADE_VERSION > values-mgmt-plane-env.yaml
      
    • Agent:
      helm show values gloo-mesh-agent/gloo-mesh-agent --version $UPGRADE_VERSION > values-data-plane-env.yaml
      
  3. Review the changelog for any Helm Changes that might require modifications to your Helm chart. For example, the following Helm chart updates might impact your upgrade from an earlier version to Gloo 2.3.22, especially if you try to reuse your existing Helm values file.

    • Sidecar acceleration: Support for the eBPF-based acceleration alpha feature is removed.
  4. Edit the Helm chart values file or prepare the --set flags to make any changes that you want. If you do not want to use certain settings, comment them out.

    For example, you might modify the following settings during an upgrade.

    • The Gloo image.tag value for the version that you want to upgrade to.
    • Istio images.hub and images.tag fields for the Solo distribution of Istio that you want to use.
    • The relay section for custom certificates to use to establish mTLS between the management server and agent.
    • The rate-limiter and ext-auth-service sections in the agent chart.

    Need to make changes after you upgrade? You can update these settings later. For more information, see Modify Helm chart values.

Step 4: Upgrade and verify the Helm installation

  1. Upgrade the Gloo Mesh Helm installation. Make sure to update any --set or --values options, such as the values file name. For example, if you installed Gloo in a single-cluster environment using meshctl, your values file might be named gs-single-values.yaml instead. Note: If the release name gloo-mgmt is not found, your Helm release might be named gloo-mesh-enterprise instead.

    helm upgrade gloo-mgmt gloo-mesh-enterprise/gloo-mesh-enterprise \
    --namespace gloo-mesh \
    --set glooMeshLicenseKey=${GLOO_MESH_LICENSE_KEY} \
    --kube-context=${MGMT_CONTEXT} \
    --version ${UPGRADE_VERSION} \
    --values values-mgmt-plane-env.yaml
    
  2. Multicluster installations: Upgrade the Helm installation in your workload cluster. Make sure to update any --set or --values options, such as the values file name. For example, if you installed Gloo in a multicluster environment using meshctl, your values file might be named gs-multi-values.yaml instead. Note: If the release name gloo-agent is not found, your Helm release might be named gloo-mesh-agent instead.

    helm upgrade gloo-agent gloo-mesh-agent/gloo-mesh-agent \
    --kube-context=${REMOTE_CONTEXT} \
    --namespace gloo-mesh \
    --version ${UPGRADE_VERSION} \
    --values values-data-plane-env.yaml
    
  3. Optional: Upgrade the Gloo external auth or rate limiting services. If you already enabled these services in the single cluster or agent installation, or you do not use these services, you can skip this step.

    helm upgrade gloo-agent-addons gloo-mesh-agent/gloo-mesh-agent \
    --kube-context=${REMOTE_CONTEXT} \
    --namespace gloo-mesh-addons \
    --version ${UPGRADE_VERSION} \
    --values values.yaml
    
  4. Multicluster installations: Repeat steps 2 - 3 for each workload cluster.

  5. Optional: Check that the Gloo management and agent resources are connected. See Step 1 of Debug the management server and relay connection.

  6. Confirm that the server components such as gloo-mesh-mgmt-server run the version that you upgraded to.

    meshctl version --kubecontext ${MGMT_CONTEXT}
    

    Example output:

       "server": [
       {
         "Namespace": "gloo-mesh",
         "components": [
           {
             "componentName": "gloo-mesh-mgmt-server",
             "images": [
                {
                 "name": "gloo-mesh-mgmt-server",
                 "domain": "gcr.io",
                 "path": "gloo-mesh-mgmt-server",
                 "version": "2.2.9"
               }
             ]
           },
       

  7. Confirm that the agent components such as gloo-mesh-agent run the version that you upgraded to.

    meshctl version --kubecontext ${REMOTE_CONTEXT}
    

    Example output:

       {
             "componentName": "gloo-mesh-agent",
             "images": [
               {
                 "name": "gloo-mesh-agent",
                 "domain": "gcr.io",
                 "path": "gloo-mesh/gloo-mesh-agent",
                 "version": "2.2.9"
               }
             ]
           },
       

Next steps

Now that you upgraded Gloo Mesh Enterprise, you must upgrade your meshctl CLI to the matching version. Depending on the Gloo Mesh Enterprise version support, you might also want to upgrade Istio or Kubernetes in your clusters.

  1. Upgrade the meshctl CLI to the same version of Gloo Mesh.
  2. Optional: If the new version of Gloo Mesh supports a more recent version of Istio, you can upgrade Istio.
  3. Optional: If the new version of Gloo Mesh supports a more recent version of Kubernetes, you can upgrade Kubernetes on your cluster. For more information, consult your cluster infrastructure provider.

Update your Gloo license

Before your Gloo license expires, you can update the license by patching the license key secret. If you use Gloo Mesh along with other Gloo products such as Gloo Gateway and Gloo Network, you can also update those licenses.

If you notice that your Gloo management component deployments are in a crash loop, your Gloo license might be expired. You can check the logs for one of the deployments, such as the management server, to look for an error message similar to the following:

kubectl logs --context $MGMT_CONTEXT -n gloo-mesh deploy/gloo-mesh-mgmt-server

{"level":"fatal","ts":1628879186.1552186,"logger":"gloo-mesh-mgmt-server","caller":"cmd/main.go:24","msg":"License is invalid or expired, crashing - license expired", ...

To update your license key in your Gloo installation:

  1. Get a new Gloo license key by contacting your account representative. If you use Gloo Mesh along with other Gloo products such as Gloo Gateway and Gloo Network, make sure to ask for up-to-date license keys for all your products.
  2. Save the new license key as an environment variable.
    export GLOO_MESH_LICENSE_KEY=<new-key-string>
    
    export GLOO_GATEWAY_LICENSE_KEY=<new-key-string>
    
    export GLOO_NETWORK_LICENSE_KEY=<new-key-string>
    
  3. Encode the license key in base64. Note: If you see the error base64: invalid option -- w, remove the -w 0 flag.
    export GLOO_MESH_LICENSE_KEY_BASE64=$(echo $GLOO_MESH_LICENSE_KEY | base64 -w 0)
    echo $GLOO_MESH_LICENSE_KEY_BASE64
    
    export GLOO_GATEWAY_LICENSE_KEY_BASE64=$(echo $GLOO_GATEWAY_LICENSE_KEY | base64 -w 0)
    echo $GLOO_GATEWAY_LICENSE_KEY_BASE64
    
    export GLOO_NETWORK_LICENSE_KEY_BASE64=$(echo $GLOO_NETWORK_LICENSE_KEY | base64 -w 0)
    echo $GLOO_NETWORK_LICENSE_KEY_BASE64
    
  4. Update your installation to use the new Gloo Mesh license key.
    kubectl patch secret license-keys -n gloo-mesh -p="{\"data\":{\"gloo-mesh-license-key\": \"$GLOO_MESH_LICENSE_KEY_BASE64\"}}" -v=1 --context ${MGMT_CONTEXT}
    
    kubectl patch secret license-keys -n gloo-mesh -p="{\"data\":{\"gloo-gateway-license-key\": \"$GLOO_GATEWAY_LICENSE_KEY_BASE64\"}}" -v=1 --context ${MGMT_CONTEXT}
    
    kubectl patch secret license-keys -n gloo-mesh -p="{\"data\":{\"gloo-network-license-key\": \"$GLOO_NETWORK_LICENSE_KEY_BASE64\"}}" -v=1 --context ${MGMT_CONTEXT}
    
  5. Optional: If your license expired and the management server pods are in a crash loop, restart the management server pods. If you updated the license before expiration, skip this step.
    kubectl rollout restart -n gloo-mesh deployment/gloo-mesh-mgmt-server --context ${MGMT_CONTEXT}