Upgrade

Upgrade minor and patch versions of Solo Enterprise for Istio.

You can use this guide to upgrade the version of your Solo Enterprise for Istio components, such as the management server and agents, or to apply changes to the components’ configuration settings.

Make sure that you review the breaking changes 🔥 that were introduced in this release and the impact that they have on your current environment. Then, continue with your upgrade.

Considerations

Consider the following rules before you plan your Solo Enterprise for Istio upgrade.

Testing upgrades

During the upgrade, the data plane continues to run, but you might not be able to modify configuration through the management plane. Because zero downtime is not guaranteed, try testing the upgrade in a staging environment before upgrading your production environment.

Patch and minor versions

Patch version upgrades: You can skip patch versions within the same minor release. For example, you can upgrade from version 2.12.0 to 2.12.2 directly, and skip the patch versions in between.

Minor version upgrades:

Always upgrade to the latest patch version of the target minor release. For example, if you want to upgrade from version 2.11.9 to 2.12.x, and 2.12.7 is the latest patch version, upgrade to that version and skip any previous patch versions for that minor release.
Do not skip minor versions during your upgrade. Upgrade minor release versions one at a time. For example, if you want to upgrade from 2.12.x to 2.14.x, you must first upgrade to the latest patch version of the 2.12 minor release. After you upgrade to 2.13.x, you can then plan your upgrade to the latest patch version of the 2.14.x release.

Multicluster only: Version skew policy for management and remote clusters

Plan to always upgrade your Solo Enterprise for Istio management server and agents to the same target version. Always upgrade the management server first. Then, roll out the upgrade to the agents in your workload clusters. During this upgrade process, your management server and agents can be one minor version apart.

For example, let’s say you want to upgrade from 2.11.9 to 2.12.x. Start by upgrading your management server to the latest patch version of the 2.12 minor release. Your management server and agent are still compliant as they are one minor version apart. Then, roll out the 2.12 minor release upgrade to the agents in your workload clusters.

If you plan to upgrade more than one minor releases, you must perform one minor release upgrade at a time. For example, to upgrade your management server and agent from 2.12.x to 2.14.x, you upgrade your management server to the latest patch version of the 2.13 minor release first. Your management server and agent are compliant because they are one minor version apart. Then, you upgrade your agents to the 2.13 minor release. After you verify the 2.13 upgrade, use the same approach to upgrade the management server and agents from 2.13 to the target 2.14 minor release.

If both your management server and agent run the same minor version, the agent can run any patch version that is equal or lower than the management server’s patch version.

Consider the following example version skew scenarios:

Supported?	Management server version	Agent version	Requirement
✅	2.12.3	2.12.1	The management server and agents run the same minor version. The agent patch version is equal to or lower than the management server.
❌	2.12.3	2.12.6	The agent runs the same minor version as the server, but has a patch version greater than the server.
✅	2.12.3	2.11.1	The agent runs a minor version no greater than n-1 behind the server.
❌	2.12.3	2.10.2	The agent runs a minor version that is greater than n-1 behind the server.

Step 1: Prepare to upgrade

Review changes from the previous version.
Caution
Before you proceed with the upgrade, make sure that you review the breaking changes that were introduced in this release and the impact that they have on your current environment. For more information, see the Breaking changes.
Check that your underlying Kubernetes platform and Istio service mesh run supported versions for the Solo Enterprise for Istio version that you want to upgrade to.
1. Review the supported versions.
2. Compare the supported version against the versions of Kubernetes and Istio that you run in your clusters.
3. If necessary, upgrade Istio or Kubernetes to a version that is supported by the Solo Enterprise for Istio version that you want to upgrade to.
  - For managed Istio installations, see Upgrade Gloo Operator-managed service meshes, or for manually installed Istio, see Upgrade Istio service meshes with Helm.
  - For Kubernetes steps, consult your cluster infrastructure provider.
Set the Solo Enterprise for Istio version that you want to upgrade to as an environment variable. The latest version is used as an example. Append -fips for a FIPS-compliant image, such as 2.12.7-fips. Do not include v before the version number.
```
export UPGRADE_VERSION=2.12.7
```

Step 2: Upgrade the `meshctl` CLI

Upgrade the meshctl CLI to the version of Solo Enterprise for Istio you want to upgrade to.

Re-install meshctl to the upgrade version.

curl -sL https://run.solo.io/meshctl/install | GLOO_MESH_VERSION=v${UPGRADE_VERSION} sh -

Verify that the client version matches the version you want to upgrade to.
```
meshctl version
```
Example output:
```
{
"client": {
  "version": "2.12.7"
},
```

Step 3: Upgrade Solo Enterprise for Istio

Upgrade your Solo Enterprise for Istio installation. The steps differ based on whether you run Solo Enterprise for Istio in a single-cluster or multicluster environment.

Single cluster

Update the gloo-platform Helm repo.

helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts
helm repo update

Apply the custom resource definitions (CRDs) for the upgrade version.

helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
    --namespace gloo-mesh \
    --version ${UPGRADE_VERSION} \
    --set installEnterpriseCrds=false

Get the Helm values file for your current version.

helm get values gloo-platform -o yaml -n gloo-mesh > gloo-single.yaml
open gloo-single.yaml

Compare your current Helm chart values with the version that you want to upgrade to. You can get a values file for the upgrade version with the helm show values command.
```
helm show values gloo-platform/gloo-platform --version ${UPGRADE_VERSION} > all-values.yaml
```
Make any changes that you want, such as modifications required for breaking changes or to enable new features, by editing your gloo-single.yaml Helm values file or preparing the --set flags. If you do not want to use certain settings, comment them out.
Upgrade the Solo Enterprise for Istio Helm installation.
Make sure to include your Helm values when you upgrade either as a configuration file in the –values flag or with –set flags. Otherwise, any previous custom values that you set might be overwritten.
```
helm upgrade gloo-platform gloo-platform/gloo-platform \
    --namespace gloo-mesh \
    -f gloo-single.yaml \
    --version ${UPGRADE_VERSION}
```

Ensure that all components are healthy and no errors are reported.

meshctl check

Example output:

🟢 CRD version check

🟢 Gloo deployment status

Namespace | Name                           | Ready | Status
gloo-mesh | gloo-mesh-ui                   | 1/1   | Healthy
gloo-mesh | prometheus-server              | 1/1   | Healthy
gloo-mesh | gloo-telemetry-collector-agent | 2/2   | Healthy

Connected Pod | Clusters

🟢 Istio compatibility check

All Istio versions found are compatible

Confirm that Solo Enterprise for Istio components, such as the gloo-mesh-ui, run the version that you upgraded to.

meshctl version

Example output:

"server": [
{
  "Namespace": "gloo-mesh",
  "components": [
    {
      "componentName": "gloo-mesh-ui",
      "images": [
         {
          "name": "gloo-mesh-ui",
          "domain": "gcr.io",
          "path": "gloo-mesh-ui",
          "version": "2.12.7"
        }
      ]
    },

Multicluster

Scale down the management server to 0 replicas. This step is required to safely enable I/O threads in Redis. If you do not want to set I/O threads in Redis, you can skip this step.
```
kubectl scale deployment gloo-mesh-mgmt-server --context ${context1} --replicas=0 -n gloo-mesh
```

Update the gloo-platform Helm repo.

helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts
helm repo update

Get the Helm values files for your current version.

Get your current values for the management plane.

helm get values gloo-platform -n gloo-mesh -o yaml --kube-context ${context1} > mgmt-plane.yaml
open mgmt-plane.yaml

Get your current values for the data plane in connected clusters.

helm get values gloo-platform -n gloo-mesh -o yaml --kube-context ${context2} > data-plane.yaml
open data-plane.yaml

Compare your current Helm chart values with the version that you want to upgrade to. You can get a values file for the upgrade version with the helm show values command.
```
helm show values gloo-platform/gloo-platform --version ${UPGRADE_VERSION} > all-values.yaml
```
Make any changes that you want, such as modifications required for breaking changes or to enable new features, by editing your mgmt-plane.yaml and data-plane.yaml Helm values files or preparing the --set flags. If you do not want to use certain settings, comment them out.

Upgrade the Helm release for the management plane.

You must always upgrade the Solo Enterprise for Istio management plane before upgrading the data plane to avoid unexpected behavior. Note that only n-1 minor version skew is supported between the management server and the agent. For more information, see the skew policy.

Apply the custom resource definitions (CRDs) for the upgrade version in the management cluster.

helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
    --kube-context ${context1} \
    --namespace gloo-mesh \
    --version ${UPGRADE_VERSION} \
    --set installEnterpriseCrds=false

Upgrade your Helm release for the management plane. Make sure to include your Helm values when you upgrade either as a configuration file in the --values flag or with --set flags. Otherwise, any previous custom values that you set might be overwritten.
```
helm upgrade gloo-platform gloo-platform/gloo-platform \
    --kube-context ${context1} \
    --namespace gloo-mesh \
    -f mgmt-plane.yaml \
    --version ${UPGRADE_VERSION}
```

Confirm that the management plane components, such as the gloo-mesh-mgmt-server, run the version that you upgraded to.

meshctl version --kubecontext ${context1}

Example output:

"server": [
{
  "Namespace": "gloo-mesh",
  "components": [
    {
      "componentName": "gloo-mesh-mgmt-server",
      "images": [
         {
          "name": "gloo-mesh-mgmt-server",
          "domain": "gcr.io",
          "path": "gloo-mesh-mgmt-server",
          "version": "2.12.7"
        }
      ]
    },

If you previously scaled down the management server, scale it back to the number of replicas that you had before. The following example uses 1 replica.
```
kubectl scale deployment gloo-mesh-mgmt-server --context ${context1} --replicas=1 -n gloo-mesh
```

Ensure that the agent connects to the management server.

meshctl check --kubecontext ${context2}

Example output:

...

🟢 Mgmt server connectivity to workload agents

Cluster  | Registered | Connected Pod
cluster2 | true       | gloo-mesh/gloo-mesh-mgmt-server-65bd557b95-v8qq6

Connected Pod                                    | Clusters
gloo-mesh/gloo-mesh-mgmt-server-65bd557b95-v8qq6 | 1

...

Open the Gloo UI and verify that none of your clusters are currently in safe mode. If a cluster is in safe more, the agent’s input snapshot is not yet populated in Redis. You see a yellow banner in the Gloo UI indicating the cluster that triggered safe mode. Wait until the input snapshot is populated and the yellow banner does not display in the Gloo UI anymore.
If a cluster remains in safe mode for a long time, review the logs of the agent by using the kubectl logs <agent-pod> -n gloo-mesh --context ${context2} command. To learn more about possible reasons for why the cluster remains in safe mode, see Indefinite safe mode.
```
meshctl dashboard --kubecontext ${context1}
```

Upgrade the Helm releases for the Solo Enterprise for Istio data plane in your workload clusters. Repeat these steps for each connected cluster, and be sure to update the cluster context each time.

Apply the custom resource definitions (CRDs) for the upgrade version in each connected cluster.

helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
    --kube-context ${context2} \
    --namespace=gloo-mesh \
    --version=${UPGRADE_VERSION} \
    --set installEnterpriseCrds=false

Upgrade your Helm release in each connected cluster. Make sure to include your Helm values when you upgrade either as a configuration file in the --values flag or with --set flags. Otherwise, any previous custom values that you set might be overwritten.
```
helm upgrade gloo-platform gloo-platform/gloo-platform \
    --kube-context ${context2} \
    --namespace gloo-mesh \
    -f data-plane.yaml \
    --version ${UPGRADE_VERSION}
```

Confirm that the data plane components, such as the gloo-mesh-agent, run the version that you upgraded to.

meshctl version --kubecontext ${context2}

Example output:

{
      "componentName": "gloo-mesh-agent",
      "images": [
        {
          "name": "gloo-mesh-agent",
          "domain": "gcr.io",
          "path": "gloo-mesh/gloo-mesh-agent",
          "version": "2.12.7"
        }
      ]
    },

Repeat these steps for each workload cluster, and be sure to update the cluster context each time.

Check that the management and agent components are connected, and that al components are healthy.

meshctl check --kubecontext ${context1}

Example output:

🟢 License status

INFO  gloo-mesh enterprise license expiration is 25 Aug 26 10:38 CDT

🟢 CRD version check

🟢 Gloo deployment status

Namespace | Name                           | Ready | Status
gloo-mesh | gloo-mesh-mgmt-server          | 1/1   | Healthy
gloo-mesh | gloo-mesh-redis                | 1/1   | Healthy
gloo-mesh | gloo-mesh-ui                   | 1/1   | Healthy
gloo-mesh | gloo-telemetry-gateway         | 1/1   | Healthy
gloo-mesh | prometheus-server              | 1/1   | Healthy
gloo-mesh | gloo-telemetry-collector-agent | 2/2   | Healthy

🟢 Mgmt server connectivity to workload agents

Cluster  | Registered | Connected Pod
cluster2 | true       | gloo-mesh/gloo-mesh-mgmt-server-65bd557b95-v8qq6

Connected Pod                                    | Clusters
gloo-mesh/gloo-mesh-mgmt-server-65bd557b95-v8qq6 | 1

🟢 Istio compatibility check

All Istio versions found are compatible

Open the Gloo UI and verify that none of your clusters are currently in safe mode. If a cluster is in safe more, the agent’s input snapshot is not yet populated in Redis. You see a yellow banner in the Gloo UI indicating the cluster that triggered safe mode. Wait until the input snapshot is populated and the yellow banner does not display in the Gloo UI anymore.
If a cluster remains in safe mode for a long time, review the logs of the agent by using the kubectl logs -n gloo-mesh –context ${context2} command. To learn more about possible reasons for why the cluster remains in safe mode, see Indefinite safe mode.
```
meshctl dashboard --kubecontext ${context1}
```

Update your Solo Enterprise for Istio license

Before your Solo license expires, you can update the license by patching the license key secret.

For example, if you notice that your management plane deployment is in a crash loop, your Solo Enterprise for Istio license might be expired. You can check the status of your license with the meshctl license check command.

To pass in a license key directly, encode the key in base64 and pass it in the --key flag. For example, to check your Solo Enterprise for Istio license key, you can run the following command:
```
meshctl license check --key $(echo ${SOLO_ISTIO_LICENSE_KEY} | base64 -w0) --context ${context1}
```
If you store your license keys in a Kubernetes secret, you can pass the secret YAML file in the --secrets-file flag instead.
```
meshctl license check --secrets-file license-keys.yaml --context ${context1}
```

Example output for an expired license:

WARNING  Your gloo-mesh license expired on 2024-01-24 19:30:53 +0100 CET. To get a new license, contact Support.
ERROR  License is expired. For more info, see https://docs.solo.io/gloo-mesh-enterprise/latest/setup/prepare/licensing/#update-licenses

To update your license key in your Solo installation:

Get a new Solo license key by contacting your account representative.
Save the new license key as an environment variable.
```
export SOLO_ISTIO_LICENSE_KEY=<new-key-string>
```
Update your license in your Solo Enterprise for Istio installation. The method depends on how you provided your license during your Helm installation.
If you provided your license key directly in your Helm values file, follow the regular upgrade steps to modify your Helm values file and replace your expired license with your new license.
If you created a secret to store your license, update the license secret to use the new license key.
kubectl -n gloo-mesh patch secret license-secret -p "stringData: { gloo-mesh-core-license-key: $SOLO_ISTIO_LICENSE_KEY }"
Optional: If your license expired and the management server pods are in a crash loop, restart the management server pods. If you updated the license before expiration, skip this step.
```
kubectl rollout restart -n gloo-mesh deployment/gloo-mesh-mgmt-server --context ${context1}
```
Verify that your license check is now valid, and no errors are reported.
- To pass in a license key directly, encode the key in base64 and pass it in the --key flag. For example, to check your Solo Enterprise for Istio license key, you can run the following command:
  meshctl license check --key $(echo ${SOLO_ISTIO_LICENSE_KEY} | base64 -w0) --context ${context1}
- If you store your license keys in a Kubernetes secret, you can pass the secret YAML file in the --secrets-file flag instead.
  meshctl license check --secrets-file license-keys.yaml --context ${context1}
Example output:
```
INFO  License key gloo-mesh-license-key for product gloo-mesh is valid. Expires at 08 Oct 24 12:31 CEST
SUCCESS  Licenses are valid
```

Upgrade Gloo Operator-managed Istio

To upgrade your Istio installations by using the Gloo Operator, see the upgrade guide for ambient mesh or sidecar mesh.

Advanced settings Uninstall

Upgrade

Considerations

Testing upgrades

Patch and minor versions

Multicluster only: Version skew policy for management and remote clusters

Step 1: Prepare to upgrade

Step 2: Upgrade the meshctl CLI

Step 3: Upgrade Solo Enterprise for Istio

Single cluster

Multicluster

Update your Solo Enterprise for Istio license

Upgrade Gloo Operator-managed Istio

Step 2: Upgrade the `meshctl` CLI