Upgrade
Upgrade minor and patch versions of Gloo Mesh Gateway.
You can use this guide to upgrade the Gloo version of your Gloo components, such as the management server and agents, or to apply changes to the components’ configuration settings.
In version 2.5, the naming convention for Envoy filters changed. In addition, the experimental environment variable EXPERIMENTAL_SEGMENT_ENVOY_FILTERS_BY_MATCHER
was removed and its functionality was promoted to standard behavior. The Gloo management server must re-create existing Envoy filters to apply these changes.
Make sure to thoroughly follow the upgrade steps in this guide to safely re-create the Envoy filters as part of your upgrade.
To learn about other breaking changes, see the Release notes.
Considerations
Consider the following rules before you plan your Gloo Mesh Gateway upgrade.
Testing upgrades
During the upgrade, the data plane continues to run, but you might not be able to modify the configurations through the management plane. Because zero downtime is not guaranteed, try testing the upgrade in a staging environment before upgrading your production environment.
Patch and minor versions
Patch version upgrades:You can skip patch versions within the same minor release. For example, you can upgrade from version 2.5.0 to 2.5.11 directly, and skip the patch versions in between.
Minor version upgrades:
- Always upgrade to the latest patch version of the target minor release. For example, if you want to upgrade from version 2.4.16 to 2.5.x, and 2.5.11 is the latest patch version, upgrade to that version and skip any previous patch versions for that minor release. Do not upgrade to a lower patch version, such as 2.5.0, 2.5.1, and so on.
- Do not skip minor versions during your upgrade. Upgrade minor release versions one at a time. For example, if you want to upgrade from 2.3.x to 2.5.x, you must first upgrade to the latest patch version of the 2.4 minor release. After you upgrade to 2.4.x, you can then plan your upgrade to the latest patch version of the 2.5.x release.
Multicluster only: Version skew policy for management and remote clusters
Plan to always upgrade your Gloo management server and agents to the same target version. Always upgrade the Gloo management server first. Then, roll out the upgrade to the Gloo agents in your workload clusters. During this upgrade process, your management server and agents can be one minor version apart.
For example, let’s say you want to upgrade from 2.4.16 to 2.5.x. Start by upgrading your management server to the latest patch version of the 2.5 minor release. Your management server and agent are still compliant as they are one minor version apart. Then, roll out the 2.5 minor release upgrade to the agents in your workload clusters.
If you plan to upgrade more than one minor releases, you must perform one minor release upgrade at a time. For example, to upgrade your management server and agent from 2.3.x to 2.5.x, you upgrade your management server to the latest patch version of the 2.4 minor release first. Your management server and agent are compliant because they are one minor version apart. Then, you upgrade your agents to the 2.4 minor release. After you verify the 2.4 upgrade, use the same approach to upgrade the management server and agents from 2.4 to the target 2.5 minor release.
If both your management server and agent run the same minor version, the agent can run any patch version that is equal or lower than the management server’s patch version.
Consider the following example version skew scenarios:
Supported? | Management server version | Agent version | Requirement |
---|---|---|---|
✅ | 2.5.9 | 2.5.3 | The management server and agents run the same minor version. The agent patch version is equal to or lower than the management server. |
❌ | 2.5.7 | 2.5.9 | The agent runs the same minor version as the server, but has a patch version greater than the server. |
✅ | 2.5.9 | 2.4.16 | The agent runs a minor version no greater than n-1 behind the server. |
❌ | 2.5.9 | 2.3.24 | The agent runs a minor version that is greater than n-1 behind the server. |
Step 1: Prepare to upgrade
Review changes from the previous version.
Make sure that you review the breaking changes 🔥 that were introduced in this release and the impact that they have on your current environment.Set a maintenance window to ensure that your environment is stable and to prevent configuration changes, including changes to Gloo Mesh Gateway or your services. In version 2.5, the naming convention for Envoy filters changed. In addition, the experimental environment variable
EXPERIMENTAL_SEGMENT_ENVOY_FILTERS_BY_MATCHER
was removed and its functionality was promoted to standard behavior. The Gloo management server must re-create existing Envoy filters to apply these changes. For more information, see Envoy filter changes.To safely upgrade and ensure existing Envoy filters are correctly re-created, the Gloo management server, and the Istio control plane istiod must temporarily be scaled down to 0 replicas. This upgrade procedure can have the following implications for your environment:
- Delayed configuration updates: During the upgrade, the Gloo management server and istiod control plane are temporarily scaled down. Because of that, the propagation of configuration changes to the sidecar or gateway proxy, such as new routing rules or security policies, is delayed. This can cause inconsistencies in traffic management and policy enforcement.
- Complex environments with long translation times: If you have a complex environment and your average translation time regularly takes more than 60 seconds, scaling down
istiod
might have unexpected impacts and delay the time for your traffic to continue as normal. - New pods cannot be added to the mesh: The Istio control plane istiod implements the sidecar injection webhook. When the control plane is scaled down, sidecar injection does not work and new pods cannot be added to the service mesh. You can manually inject sidecars into your pods. However, keep in mind that these pods do not receive traffic as endpoint discovery is also disabled when the Istio control plane is scaled down. After the control plane is scaled back up, pods are automatically injected with sidecars and added to the mesh.
- mTLS certificate issues: If certificates expire while the Istio control plane is not available, mutual TLS between services in the mesh might be impacted.
Review your Istio mTLS certificates and their rotation policy. Ensure that no TLS certificates expire during the upgrade.
Check that your underlying Kubernetes platform and Istio service mesh run supported versions for the Gloo Mesh Gateway version that you want to upgrade to.
- Review the supported versions.
- Compare the supported version against the versions of Kubernetes and Istio that you run in your clusters.
- If necessary, upgrade Istio or Kubernetes to a version that is supported by the Gloo Mesh Gateway version that you want to upgrade to.
- For managed Istio installations, see Upgrade managed gateways within your gloo-platform Helm chart or Upgrade gateways with lifecycle manager CRs, or for manually installed Istio, see the Istio documentation.
- For Kubernetes steps, consult your cluster infrastructure provider.
Set the Gloo Mesh Gateway version that you want to upgrade to as an environment variable. The latest version is used as an example. You can find other versions in the changelog documentation. Append
-fips
for a FIPS-compliant image, such as2.5.11-fips
. Do not includev
before the version number.Important: If you install thegloo-mesh-mgmt-server
management plane orgloo-mesh-agent
data plane in a custom namespace other thangloo-mesh
, skip upgrades to 2.5.7 and 2.5.8. These patch versions have a bug that causes IssuedCertificates to reconcile indefinitely. Instead, upgrade directly to patch version 2.5.9 or later, in which the issue is resolved.export UPGRADE_VERSION=2.5.11
Step 2: Upgrade the meshctl
CLI
Upgrade the meshctl
CLI to the version of Gloo Mesh Gateway you want to upgrade to.
Re-install
meshctl
to the upgrade version.curl -sL https://run.solo.io/meshctl/install | GLOO_MESH_VERSION=v$UPGRADE_VERSION sh -
Verify that the client version matches the version you installed.
meshctl version
Example output:
{ "client": { "version": "2.6.6" },
Step 3: Upgrade Gloo Mesh Gateway
Upgrade your Gloo Mesh Gateway installation. The steps differ based on whether you run Gloo Mesh Gateway in a single-cluster or multicluster environment.
Single cluster
Verify that your environment is stable.
Verify your Gloo Mesh Gateway setup. If errors are returned, make sure that you resolve them before you continue with your upgrade. You can use the Troubleshooting guides to debug your setup.
meshctl check
Open the Prometheus expression browser and monitor the
object_write_fails_total
metric. In the case that the Gloo management server cannot write configuration to Istio, the error is captured and the counter of this metric is increased. Before you upgrade, make sure that this metric is stable and does not increase. If you see this metric increasing, check the configuration that causes the issue and resolve it before you continue with your upgrade.meshctl proxy prometheus
Check the status of your Istio proxies and verify that all proxies show a
SYNCED
orNOT_SENT
status. If you see aSTALE
status, check the Istio logs to find the configuration that is rejected by Envoy. For example, you might have provided an invalid transformation template or WAF rule set.istioctl proxy-status
Get input and output snapshots for your cluster.
Forward port 9091 of the
gloo-mesh-mgmt-server
pod to yourlocalhost
.kubectl port-forward -n gloo-mesh deploy/gloo-mesh-mgmt-server 9091
Take snapshots in case you want to refer to the logs later, such as to open a Support issue.
curl localhost:9091/snapshots/input -o input_snapshot.json curl localhost:9091/snapshots/output -o output_snapshot.json
Get the number of Envoy filters that are applied in your cluster. While you cannot predict the number of Envoy filters that are created for you as part of the upgrade, it serves as a reference point in later steps.
kubectl get envoyfilters -A --no-headers | wc -l
Scale down the Gloo management server to 0 replicas. This step is required to safely enable safe mode and I/O threads in Redis. If you already enabled safe mode and you do not want to set I/O threads in Redis, you can skip this step.
kubectl scale deployment gloo-mesh-mgmt-server --replicas=0 -n gloo-mesh
Scale down the Istio control plane istiod.
If you have a complex environment and your average translation time regularly takes more than 60 seconds, scaling downistiod
might have unexpected impacts and delay the time for your traffic to continue as normal. As such, you might decide to skip scaling downistiod
and handle the impact of any upgrade discrepancies, such as temporary duplicate resources, yourself.Istio lifecycle manager installations only: Check the Istio lifecycle manager resource and verify that the
defaultRevision
field is set tofalse
. If thedefaultRevision
field is set totrue
, edit the Istio lifecycle manager resource and set that field tofalse
.kubectl edit istiolifecyclemanager gloo-platform -n gloo-mesh
Scale down the istiod pods to 0 replicas.
export REVISION=$(kubectl get pod -L app=istiod -n istio-system -o jsonpath='{.items[0].metadata.labels.istio\.io/rev}') kubectl scale deployment -n istio-system istiod-$REVISION --replicas 0
Verify that the istiod pods are scaled down.
kubectl get pods -n istio-system | grep istiod
Example output:
No resources found in istio-system namespace.
Get the Istio validating webhook configurations in your cluster.
kubectl get validatingwebhookconfigurations -A | grep validator
Look for any non-revisioned validating webhook configurations and update the
FailurePolicy
toIgnore
. For example, you might have agm-istio-validator
oristiod-default-validator
webhook configuration that must be updated. When you scale up the Gloo agent again, the agent writes non-revisioned Istio resources to the cluster. To allow the agent to write these resources without rejecting them, theFailurePolicy
in the non-revisioned validating webhook configurations must be set toIgnore
. Keep in mind that if you only have revisioned webhook configurations, such asistio-validator-1-18-istio-system
, no update is needed.To confirm that you have a non-revisioned validating webhook configuration, you can check thewebhook.objectSelector
field in your webhook configuration resource.kubectl patch validatingwebhookconfigurations <webhook-name> --type='json' -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value":"Ignore"}]'
Update the
gloo-platform
Helm repo.helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts helm repo update
Apply the Gloo custom resource definitions (CRDs) for the upgrade version.
helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \ --namespace gloo-mesh \ --version $UPGRADE_VERSION \ --set installEnterpriseCrds=true
Get the Helm values file for your current version.
helm get values gloo-platform -o yaml -n gloo-mesh > gloo-single.yaml open gloo-single.yaml
Compare your current Helm chart values with the version that you want to upgrade to. You can get a values file for the upgrade version with the
helm show values
command.helm show values gloo-platform/gloo-platform --version $UPGRADE_VERSION > all-values.yaml
Make any changes that you want, such as modifications required for breaking changes or to enable new features, by editing your
gloo-single.yaml
Helm values file or preparing the--set
flags. If you do not want to use certain settings, comment them out.If you have not done so already, enable safe mode on the Gloo management server. Safe mode ensures that output snapshots are only created and sent to the Gloo agents when the input snapshots of all registered Gloo agents are present in Redis or the Gloo management server cache. For more information about this feature, see safe mode.
glooMgmtServer: safeMode: true
Upgrade the Gloo Mesh Gateway Helm installation.
Make sure to include your Helm values when you upgrade either as a configuration file in the
–values
flag or with–set
flags. Otherwise, any previous custom values that you set might be overwritten.helm upgrade gloo-platform gloo-platform/gloo-platform \ --namespace gloo-mesh \ -f gloo-single.yaml \ --version $UPGRADE_VERSION
Confirm that Gloo components, such as the
gloo-mesh-mgmt-server
, run the version that you upgraded to.meshctl version
Example output:
"server": [ { "Namespace": "gloo-mesh", "components": [ { "componentName": "gloo-mesh-mgmt-server", "images": [ { "name": "gloo-mesh-mgmt-server", "domain": "gcr.io", "path": "gloo-mesh-mgmt-server", "version": "2.5.11" } ] },
If you previously scaled down the Gloo management server, scale it back to the number of replicas that you had before. The following example uses 1 replica.
kubectl scale deployment gloo-mesh-mgmt-server --replicas=1 -n gloo-mesh
Wait until all Gloo agents are connected to the Gloo management server.
meshctl check
Monitor the number of Envoy filter resources in the cluster and compare it to the number that you noted earlier. If the re-creation of Envoy filters is successful, you see an increased number of Envoy filters in your cluster. Note that depending on the size of your environment, this process might take a few minutes to complete. As soon as the number of Envoy filters stabilizes and does not increase anymore, the re-creation of Envoy filters is done and you can continue with scaling istiod back up.
kubectl get envoyfilters -A --no-headers | wc -l
Open the Prometheus expression browser and ensure that the
object_write_fails_total
metric is stable and does not increase.meshctl proxy prometheus open http://localhost:9091
Scale up istiod.
kubectl scale deployment -n istio-system istiod-$REVISION --replicas 1
Confirm that any revisionless validating webhook configurations that you previously updated show a
FailurePolicy
ofFail
again. If not, update the failure policy and set it toFail
.kubectl get validatingwebhookconfigurations <webhook-name> -o yaml | grep "failurePolicy" kubectl patch validatingwebhookconfigurations <webhook-name> --type='json' -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value":"Fail"}]'
Istio lifecycle manager installations only: If you previously changed the
defaultRevision
field in the Istio lifecycle manager resource, edit the resource and set thedefaultRevision
field back totrue
.kubectl get istiolifecyclemanager gloo-platform -n gloo-mesh -o yaml kubectl edit istiolifecyclemanager gloo-platform -n gloo-mesh
Multicluster
Verify that your environment is stable.
Verify your Gloo Mesh Gateway setup. If errors are returned, make sure that you resolve them before you continue with your upgrade. You can use the Troubleshooting guides to debug your setup.
meshctl check --kubecontext $MGMT_CONTEXT meshctl check --kubecontext $REMOTE_CONTEXT
Open the Prometheus expression browser and monitor the
object_write_fails_total
metric. In the case that the Gloo management server cannot write configuration to Istio, the error is captured and the counter of this metric is increased. Before you upgrade, make sure that this metric is stable and does not increase. If you see this metric increasing, check the configuration that causes the issue and resolve it before you continue with your upgrade.meshctl proxy prometheus --kubecontext $MGMT_CONTEXT
Check the status of your Istio proxies and verify that all proxies show a
SYNCED
orNOT_SENT
status. If you see aSTALE
status, check the Istio logs to find the configuration that is rejected by Envoy. For example, you might have provided an invalid transformation template or WAF rule set.istioctl proxy-status --context $REMOTE_CONTEXT
Get input and output snapshots for all your workload clusters.
Forward port 9091 of the
gloo-mesh-mgmt-server
pod to yourlocalhost
.kubectl port-forward --context $MGMT_CONTEXT -n gloo-mesh deploy/gloo-mesh-mgmt-server 9091
Take snapshots in case you want to refer to the logs later, such as to open a Support issue.
curl localhost:9091/snapshots/input -o input_snapshot.json curl localhost:9091/snapshots/output -o output_snapshot.json
Get the number of Envoy filters that are applied in each workload cluster. While you cannot predict the number of Envoy filters that are created for you as part of the upgrade, it serves as a reference point in later steps.
kubectl get envoyfilters --context $REMOTE_CONTEXT -A --no-headers | wc -l
Scale down the Gloo management server to 0 replicas. This step is required to safely enable safe mode and I/O threads in Redis. If you already enabled safe mode and you do not want to set I/O threads in Redis, you can skip this step.
kubectl scale deployment gloo-mesh-mgmt-server --context $MGMT_CONTEXT --replicas=0 -n gloo-mesh
For each workload cluster, scale down the Istio control plane istiod.
If you have a complex environment and your average translation time regularly takes more than 60 seconds, scaling downistiod
might have unexpected impacts and delay the time for your traffic to continue as normal. As such, you might decide to skip scaling downistiod
and handle the impact of any upgrade discrepancies, such as temporary duplicate resources, yourself.Istio lifecycle manager installations only: Check the Istio lifecycle manager resource and verify that the
defaultRevision
field is set tofalse
. If thedefaultRevision
field is set totrue
, edit the Istio lifecycle manager resource and set that field tofalse
.kubectl get IstioLifecycleManager istiod-control-plane -n gloo-mesh --context $MGMT_CONTEXT -o yaml kubectl edit IstioLifecycleManager istiod-control-plane -n gloo-mesh --context $MGMT_CONTEXT
Scale down the istiod pods to 0 replicas.
export REVISION=$(kubectl get pod -L app=istiod -n istio-system --context $REMOTE_CONTEXT -o jsonpath='{.items[0].metadata.labels.istio\.io/rev}') kubectl scale deployment -n istio-system istiod-$REVISION --replicas 0 --context $REMOTE_CONTEXT
Verify that the istiod pods are scaled down.
kubectl get pods -n istio-system --context $REMOTE_CONTEXT | grep istiod
Example output:
No resources found in istio-system namespace.
Get the Istio validating webhook configurations in your cluster.
kubectl get validatingwebhookconfigurations -A --context $REMOTE_CONTEXT | grep validator
Look for any non-revisioned validating webhook configurations and update the
FailurePolicy
toIgnore
. For example, you might have agm-istio-validator
oristiod-default-validator
webhook configuration that must be updated. When you scale up the Gloo agent again, the agent writes non-revisioned Istio resources to the cluster. To allow the agent to write these resources without rejecting them, theFailurePolicy
in the non-revisioned validating webhook configurations must be set toIgnore
. Keep in mind that if you only have revisioned webhook configurations, such asistio-validator-1-18-istio-system
, no update is needed.To confirm that you have a non-revisioned validating webhook configuration, you can check thewebhook.objectSelector
field in your webhook configuration resource.kubectl get validatingwebhookconfiguration <webhook-name> -o yaml | grep failurePolicy kubectl patch validatingwebhookconfigurations <webhook-name> --context $REMOTE_CONTEXT--type='json' -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value":"Ignore"}]'
Update the
gloo-platform
Helm repo.helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts helm repo update
Get the Helm values files for your current version.
- Get your current values for the management cluster.
helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $MGMT_CONTEXT > mgmt-plane.yaml open mgmt-plane.yaml
- Get your current values for the workload clusters.
helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $REMOTE_CONTEXT > data-plane.yaml open data-plane.yaml
- Optional: If you maintain a separate
gloo-agent-addons
Helm release, get the values for that Helm release too. Note that your add-ons might be installed in a different namespace, such asgloo-mesh-addons
.helm get values gloo-agent-addons -n gloo-mesh -o yaml --kube-context $REMOTE_CONTEXT > gloo-agent-addons.yaml open gloo-agent-addons.yaml
- Get your current values for the management cluster.
Compare your current Helm chart values with the version that you want to upgrade to. You can get a values file for the upgrade version with the
helm show values
command.helm show values gloo-platform/gloo-platform --version $UPGRADE_VERSION > all-values.yaml
Make any changes that you want, such as modifications required for breaking changes or to enable new features, by editing your
mgmt-plane.yaml
anddata-plane.yaml
Helm values files or preparing the--set
flags. If you do not want to use certain settings, comment them out.If you have not done so already, enable safe mode in the Helm values file for the Gloo management cluster. Safe mode ensures that output snapshots are only created and sent to the Gloo agents when the input snapshots of all registered Gloo agents are present in Redis or the Gloo management server cache. For more information about this feature, see safe mode.
glooMgmtServer: safeMode: true
Upgrade the Gloo Mesh Gateway Helm release in your management cluster.
You must always upgrade the Gloo management plane before upgrading the Gloo data plane to avoid unexpected behavior. Note that only
n-1
minor version skew is supported between the Gloo management server and the agent. For more information, see the skew policy.- Apply the Gloo custom resource definitions (CRDs) for the upgrade version in the management cluster.
helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \ --kube-context $MGMT_CONTEXT \ --namespace gloo-mesh \ --version $UPGRADE_VERSION \ --set installEnterpriseCrds=true
- Upgrade your Helm release in the management cluster. Make sure to include your Helm values when you upgrade either as a configuration file in the
--values
flag or with--set
flags. Otherwise, any previous custom values that you set might be overwritten.helm upgrade gloo-platform gloo-platform/gloo-platform \ --kube-context $MGMT_CONTEXT \ --namespace gloo-mesh \ -f mgmt-plane.yaml \ --version $UPGRADE_VERSION
- Confirm that the management plane components, such as the
gloo-mesh-mgmt-server
, run the version that you upgraded to.Example output:meshctl version --kubecontext $MGMT_CONTEXT
"server": [ { "Namespace": "gloo-mesh", "components": [ { "componentName": "gloo-mesh-mgmt-server", "images": [ { "name": "gloo-mesh-mgmt-server", "domain": "gcr.io", "path": "gloo-mesh-mgmt-server", "version": "2.5.11" } ] },
- Apply the Gloo custom resource definitions (CRDs) for the upgrade version in the management cluster.
If you previously scaled down the Gloo management server, scale it back to the number of replicas that you had before. The following example uses 1 replica.
kubectl scale deployment gloo-mesh-mgmt-server --context $MGMT_CONTEXT --replicas=1 -n gloo-mesh
Wait until all Gloo agents are connected to the Gloo management server.
meshctl check --kubecontext $REMOTE_CONTEXT
Open the Gloo UI and verify that none of your clusters are currently in safe mode. If a cluster is in safe more, the agent’s input snapshot is not yet populated in Redis. You see a yellow banner in the Gloo UI indicating the cluster that triggered safe mode. Wait until the input snapshot is populated and the yellow banner does not display in the Gloo UI anymore.
If a cluster remains in safe mode for a long time, review the logs of the agent by using thekubectl logs <agent-pod> -n gloo-mesh --context $REMOTE_CONTEXT
command. To learn more about possible reasons for why the cluster remains in safe mode, see Indefinite safe mode.meshctl dashboard --kubecontext $MGMT_CONTEXT
Monitor the number of Envoy filter resources in the cluster and compare it to the number that you noted earlier. If the re-creation of Envoy filters is successful, you see an increased number of Envoy filters in your cluster. Note that depending on the size of your environment, this process might take a few minutes to complete. As soon as the number of Envoy filters stabilizes and does not increase anymore, the re-creation of Envoy filters is done and you can continue with upgrading the workload clusters.
kubectl get envoyfilters -A --context $REMOTE_CONTEXT --no-headers | wc -l
Open the Prometheus expression browser and ensure that the
object_write_fails_total
metric is stable and does not increase.meshctl proxy prometheus --kubecontext $MGMT_CONTEXT open http://localhost:9091
Upgrade the Gloo Mesh Gateway Helm releases in your workload clusters. Repeat these steps for each workload cluster, and be sure to update the cluster context each time.
Apply the Gloo custom resource definitions (CRDs) for the upgrade version in each workload cluster.
helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \ --kube-context $REMOTE_CONTEXT \ --namespace=gloo-mesh \ --version=$UPGRADE_VERSION \ --set installEnterpriseCrds=true
Upgrade your Helm release in each workload cluster. Make sure to include your Helm values when you upgrade either as a configuration file in the
--values
flag or with--set
flags. Otherwise, any previous custom values that you set might be overwritten.helm upgrade gloo-platform gloo-platform/gloo-platform \ --kube-context $REMOTE_CONTEXT \ --namespace gloo-mesh \ -f data-plane.yaml \ --version $UPGRADE_VERSION
Optional: If you maintain a separate
gloo-agent-addons
Helm release, upgrade that Helm release in each workload cluster too. Be sure to update the cluster context for each workload cluster that you repeat this command for. Note that your add-ons might be installed in a different namespace, such asgloo-mesh-addons
.helm upgrade gloo-agent-addons gloo-platform/gloo-platform \ --kube-context $REMOTE_CONTEXT \ --namespace gloo-mesh \ -f gloo-agent-addons.yaml \ --version $UPGRADE_VERSION
Ensure that the Gloo agent connects to the Gloo management server.
meshctl check --kubecontext $REMOTE_CONTEXT
Open the Gloo UI and verify that none of your clusters are currently in safe mode. If a cluster is in safe more, the agent’s input snapshot is not yet populated in Redis. You see a yellow banner in the Gloo UI indicating the cluster that triggered safe mode. Wait until the input snapshot is populated and the yellow banner does not display in the Gloo UI anymore.
If a cluster remains in safe mode for a long time, review the logs of the agent by using thekubectl logs <agent-pod> -n gloo-mesh --context $REMOTE_CONTEXT
command. To learn more about possible reasons for why the cluster remains in safe mode, see Indefinite safe mode.meshctl dashboard --kubecontext $MGMT_CONTEXT
Confirm that the data plane components, such as the
gloo-mesh-agent
, run the version that you upgraded to.meshctl version --kubecontext $REMOTE_CONTEXT
Example output:
{ "componentName": "gloo-mesh-agent", "images": [ { "name": "gloo-mesh-agent", "domain": "gcr.io", "path": "gloo-mesh/gloo-mesh-agent", "version": "2.5.11" } ] },
Repeat these steps for each workload cluster, and be sure to update the cluster context each time.
Scale up istiod in all workload clusters.
kubectl scale deployment -n istio-system --context $REMOTE_CONTEXT istiod-$REVISION --replicas 1
Confirm that any revisionless validating webhook configurations that you previously updated show a
FailurePolicy
ofFail
again. If not, update the failure policy and set it toFail
.kubectl get validatingwebhookconfigurations <webhook-name> --context $REMOTE_CONTEXT -o yaml | grep "failurePolicy" kubectl patch validatingwebhookconfigurations <webhook-name> --context $REMOTE_CONTEXT --type='json' -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value":"Fail"}]'
Istio lifecycle manager installations only: If you previously changed the
defaultRevision
field in the Istio lifecycle manager resource, edit the resource and set thedefaultRevision
field back totrue
.kubectl edit istiolifecyclemanager gloo-platform --context $MGMT_CONTEXT -n gloo-mesh
Update your Gloo license
Before your Gloo license expires, you can update the license by patching the license key secret. If you use Gloo Mesh along with other Gloo products such as Gloo Mesh Gateway, you can also update those licenses.
For example, if you notice that your Gloo management plane deployments are in a crash loop, your Gloo license might be expired. You can check the status of your license with the meshctl license check
command.
- To pass in a license key directly, encode the key in base64 and pass it in the
--key
flag. For example, to check your Gloo Mesh Enterprise license key, you can run the following command:meshctl license check --key $(echo ${GLOO_MESH_LICENSE_KEY} | base64 -w0) --context ${MGMT_CONTEXT}
- If you store your license keys in a Kubernetes secret, you can pass the secret YAML file in the
--secrets-file
flag instead.meshctl license check --secrets-file license-keys.yaml --context ${MGMT_CONTEXT}
Example output for an expired license:
WARNING Your gloo-mesh license expired on 2024-01-24 19:30:53 +0100 CET. To get a new license, contact Support.
ERROR License is expired. For more info, see https://docs.solo.io/gloo-mesh-enterprise/latest/setup/prepare/licensing/#update-licenses
To update your license key in your Gloo installation:
Get a new Gloo license key by contacting your account representative. If you use Gloo Mesh along with other Gloo products such as Gloo Mesh Gateway, make sure to ask for up-to-date license keys for all your products.
Save the new license key as an environment variable.
Update the license secret to use the new Gloo license key.
Optional: If your license expired and the management server pods are in a crash loop, restart the management server pods. If you updated the license before expiration, skip this step.
kubectl rollout restart -n gloo-mesh deployment/gloo-mesh-mgmt-server --context ${MGMT_CONTEXT}
Verify that your license check is now valid, and no errors are reported.
- To pass in a license key directly, encode the key in base64 and pass it in the
--key
flag. For example, to check your Gloo Mesh Enterprise license key, you can run the following command:meshctl license check --key $(echo ${GLOO_MESH_LICENSE_KEY} | base64 -w0) --context ${MGMT_CONTEXT}
- If you store your license keys in a Kubernetes secret, you can pass the secret YAML file in the
--secrets-file
flag instead.meshctl license check --secrets-file license-keys.yaml --context ${MGMT_CONTEXT}
Example output:
INFO License key gloo-mesh-license-key for product gloo-mesh is valid. Expires at 08 Oct 24 12:31 CEST SUCCESS Licenses are valid
- To pass in a license key directly, encode the key in base64 and pass it in the
Upgrade managed gateways within your gloo-platform Helm chart
The steps in this section upgrade Istio installations that are managed by the istioInstallations
section of your Gloo Platform Helm chart. If you directly created IstioLifecycleManager
and GatewayLifecycleManager
CRs to manage the Istio control planes and gateways instead of using the Helm chart option, see Upgrade managed gateways with IstioLifecycleManager
and GatewayLifecycleManager
CRs instead.
If you manage Istio installations within the istioInstallations
section your Gloo Platform Helm chart, you can apply updates to your Istio installations in one of the following ways:
- In-place upgrade
- Revisioned canary upgrade
- Testing only: Manually replacing the IstioLifecycleManager CR
In-place upgrades
Istio 1.21 is not supported in Gloo Mesh Gateway version 2.5.
Istio 1.20 is supported only as patch version 1.20.1-patch1
and later. Do not use patch versions 1.20.0 and 1.20.1, which contain bugs that impact several Gloo Mesh Gateway features that rely on Istio ServiceEntries.
If you have multiple external services that use the same host and plan to use Istio 1.20, you must use patch version 1.20.7 or later to ensure that the Istio service entry that is created for those external services is correct.
In an in-place upgrade, Gloo upgrades your existing control plane or gateway installations. Note that in production environments, canary upgrades are recommended for updating the minor version. However, you might want to use in-place upgrades for patch versions or changes within the same minor version. Be sure to test in-place upgrades in development or staging environments first.
In-place upgrades behave differently based on whether your installation is revisionless or revisioned.
Revisionless installations (testing only): If your testing-only installation is revisionless (your settings omit the revision
and gatewayRevision
fields), in-place upgrades are triggered when you apply changes to any field in the istioInstallations
section.
Revisioned installations: If your installation is revisioned, in-place upgrades are triggered only when you apply changes to one of the following fields in the istioInstallations
section. Otherwise, a canary upgrade is required.
istioOperatorSpec.tag
patch version- In-place upgrades are not supported for downgrading the patch version.
- In-place upgrades are not supported if you do not already specify a
tag
value, such as if you want to switch from theauto
setting to a specific version. This is because you must also specifyhub
andrevision
values, which require a canary upgrade.
istioOperatorSpec.meshConfig
values
To trigger an in-place upgrade:
Follow the steps in this guide to perform a regular upgrade of your Gloo Mesh installation and include your Istio changes in your Helm values file. For example, in a single-cluster setup, you might edit your Helm values file to update the patch version of Istio in the
istioInstallations.controlPlane.installations.istioOperatorSpec.tag
andistioInstallations.northSouthGateways.installations.istioOperatorSpec.tag
fields. After you apply the updates in your Helm upgrade of thegloo-platform
chart, Gloo starts an in-place upgrade of the Istio control plane and gateways.Required for versions earlier than 2.5.9, 2.4.16, and 2.3.24: After your Helm upgrade completes, restart your gateway pods in each workload cluster. For example, you might use the following command to rollout a restart of the
istio-eatwestgateway-1-20
andistio-ingressgateway-1-20
deployments.
kubectl rollout restart -n gloo-mesh-gateways deployment/istio-eastwestgateway-1-20 --context $REMOTE_CONTEXT1
kubectl rollout restart -n gloo-mesh-gateways deployment/istio-ingressgateway-1-20 --context $REMOTE_CONTEXT1
kubectl rollout restart -n gloo-mesh-gateways deployment/istio-eastwestgateway-1-20 --context $REMOTE_CONTEXT2
kubectl rollout restart -n gloo-mesh-gateways deployment/istio-ingressgateway-1-20 --context $REMOTE_CONTEXT2
Revisioned canary upgrades
Istio 1.21 is not supported in Gloo Mesh Gateway version 2.5.
Istio 1.20 is supported only as patch version 1.20.1-patch1
and later. Do not use patch versions 1.20.0 and 1.20.1, which contain bugs that impact several Gloo Mesh Gateway features that rely on Istio ServiceEntries.
If you have multiple external services that use the same host and plan to use Istio 1.20, you must use patch version 1.20.7 or later to ensure that the Istio service entry that is created for those external services is correct.
In a canary upgrade, you install another Istio installation (canary) alongside your active installation. Each installation is revisioned so that you can easily identify and verify the separate settings and resources for each installation. Note that during a canary upgrade, the validating admissions webhook is enabled only for the canary installation to prevent issues that occur when multiple webhooks are enabled.
Perform a canary upgrade when you change one of the following fields:
istioOperatorSpec.tag
minor versionistioOperatorSpec.hub
repository, such as switching to the repository for the minor version of the Solo distribution of Istio that you want to upgrade tocomponents
,profile
,values
, ornamespace
in theistioOperatorSpec
To perform a canary upgrade:
OpenShift only: Elevate the permissions of the service account that will be created for the new revision’s operator project. This permission allows the Istio sidecar to make use of a user ID that is normally restricted by OpenShift. Replace the revision with the revision you plan to use.
oc adm policy add-scc-to-group anyuid system:serviceaccounts:gm-iop-1-20 --context $REMOTE_CONTEXT1 oc adm policy add-scc-to-group anyuid system:serviceaccounts:gm-iop-1-20 --context $REMOTE_CONTEXT2
Follow the steps in this guide to perform a regular upgrade of your Gloo Mesh installation. When you edit the
istioInstallations.controlPlane
andistioInstallations.eastWestGateways
sections of your Helm values file, add another installation entry for the canary revision, and leave the entry your current installation as-is. For the canary revision, be sure to setdefaultRevision
andactiveGateway
tofalse
so that only the existing revisions continue to run.For example, you might add the following installation entries for the Istio control plane and east-west gateway alongside your existing entries. If you have a Gloo Mesh Gateway license, you might also have entries for the ingress gateway proxy in thenothSouthGateways
section too.istioInstallations: controlPlane: enabled: true installations: # EXISTING revision - clusters: - defaultRevision: true # Keep this field set to TRUE name: cluster1 trustDomain: "" istioOperatorSpec: hub: $REPO tag: 1.19.10-solo profile: minimal namespace: istio-system ... revision: 1-19 # NEW revision - clusters: - defaultRevision: false # Set this field to FALSE name: cluster1 trustDomain: "" istioOperatorSpec: hub: $REPO tag: 1.20.8-patch1-solo profile: minimal namespace: istio-system ... revision: 1-20 eastWestGateways: - enabled: true installations: # EXISTING revision - clusters: - activeGateway: true # Keep this field set to TRUE name: cluster1 name: trustDomain: "" gatewayRevision: 1-19 istioOperatorSpec: hub: $REPO tag: 1.19.10-solo profile: empty namespace: gloo-mesh-gateways ... # NEW revision - clusters: - defaultRevision: false # Set this field to FALSE name: cluster1 name: trustDomain: "" gatewayRevision: 1-20 istioOperatorSpec: hub: $REPO tag: 1.20.8-patch1-solo profile: empty namespace: gloo-mesh-gateways ... name: istio-eastwestgateway enabled: true
- Updating the minor version of Istio? In your canary revision section, be sure to update both the repo key in the
hub
field, and the Istio version in thetag
field. You can get the repo key for the Istio version that you want to install from the Istio images built by Solo.io support article. - For most use cases, you can set the
revision
and thegatewayRevision
to the same version. However, gateway installations can point to anyistiod
control plane revision by using thecontrolPlaneRevision
field. For simplicity, if you do not specifycontrolPlaneRevision
, the gateway installation uses a control plane with the same revision as itself.
- Updating the minor version of Istio? In your canary revision section, be sure to update both the repo key in the
After you apply the Helm upgrade with your updated values file, verify that Istio resources for the canary installation are created. For example, if you updated the Istio minor version to
1-20
, verify that resources are created in thegm-iop-1-20
namespace, and that resources for1-20
are created alongside the existing resources for the previous version in theistio-system
andgloo-mesh-gateways
namespaces. Note that the gateway load balancers for the canary revision contain the revision in the name, such asistio-eastwestgateway-1-20
.kubectl get all -n gm-iop-1-20 --context $REMOTE_CONTEXT kubectl get all -n istio-system --context $REMOTE_CONTEXT kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT
After performing any necessary testing, switch to the new Istio control plane and gateway revisions.
- Get your Helm values file. Change the release name as needed.
helm get values gloo-platform -n gloo-mesh -o yaml > mgmt-plane.yaml open mgmt-plane.yaml
- Change
defaultRevision
andactiveGateway
tofalse
for the old revision and totrue
for the new revision.- New load balancers are created for the canary gateways. To instead change the control plane revision in use by the existing gateway load balancers, you can set the
istio.io/rev
label on the gateway deployment, which triggers a rolling restart.
istioInstallations: controlPlane: enabled: true installations: # EXISTING revision - clusters: - defaultRevision: false # Set this field to FALSE name: cluster1 trustDomain: "" istioOperatorSpec: hub: $REPO tag: 1.19.10-solo profile: minimal namespace: istio-system ... revision: 1-19 # NEW revision - clusters: - defaultRevision: true # Set this field to TRUE name: cluster1 trustDomain: "" istioOperatorSpec: hub: $REPO tag: 1.20.8-patch1-solo profile: minimal namespace: istio-system ... revision: 1-20 eastWestGateways: - enabled: true installations: # EXISTING revision - clusters: - activeGateway: false # Set this field set to FALSE name: cluster1 name: trustDomain: "" gatewayRevision: 1-19 istioOperatorSpec: hub: $REPO tag: 1.19.10-solo profile: empty namespace: gloo-mesh-gateways ... # NEW revision - clusters: - defaultRevision: true # Set this field to TRUE name: cluster1 name: trustDomain: "" gatewayRevision: 1-20 istioOperatorSpec: hub: $REPO tag: 1.20.8-patch1-solo profile: empty namespace: gloo-mesh-gateways ... name: istio-eastwestgateway enabled: true
- New load balancers are created for the canary gateways. To instead change the control plane revision in use by the existing gateway load balancers, you can set the
- Upgrade your Helm release. Change the release name as needed.
helm upgrade gloo-platform gloo-platform/gloo-platform \ --namespace gloo-mesh \ -f mgmt-plane.yaml \ --version $UPGRADE_VERSION
- Get your Helm values file. Change the release name as needed.
After your Helm upgrade completes, verify that the active gateways for the new revision are created, which do not have the revision appended to the name. Note that gateways for the inactive revision that you previously ran also exist in the namespace, in the case that a rollback is required.
kubectl get all -n gloo-mesh-gateways
Example output, in which the active gateway (
istio-eastwestgateway
) for the new revision and inactive gateway (such asistio-eastwestgateway-1-19
) for the old revision are created:NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) istio-eastwestgateway LoadBalancer 10.44.4.140 34.150.235.221 15021:31321/TCP,80:32525/TCP,443:31826/TCP 48s AGE istio-eastwestgateway-1-19 LoadBalancer 10.56.15.36 34.145.163.61 15021:31936/TCP,80:30196/TCP,443:32286/TCP,15443:31851/TCP 45s
Upgrade your apps’ Istio sidecars.
- For any namespace where you run Istio-managed apps, change the label to use the new revision. For example, you might update the
bookinfo
namespace for the canary revision1-20
. If you did not previously use revision labels for your apps, you can upgrade your application’s sidecars by runningkubectl label ns bookinfo istio-injection-
andkubectl label ns bookinfo istio.io/rev=<revision>
.- Single cluster setup:
kubectl label ns bookinfo --overwrite istio.io/rev=1-20
- Multicluster setup:
kubectl label ns bookinfo --overwrite istio.io/rev=1-20 --context $REMOTE_CONTEXT1 kubectl label ns bookinfo --overwrite istio.io/rev=1-20 --context $REMOTE_CONTEXT2
- Single cluster setup:
- Update any Istio-managed apps by rolling out restarts. The Istio sidecars for each microservice are updated to use the new Istio version. Make sure that you only restart one microservice at a time. For example, in the following commands to update the Bookinfo microservices, 20 seconds elapse between each restart to ensure that the pods have time to start running.
- Single cluster setup:
kubectl rollout restart deployment -n bookinfo details-v1 sleep 20s kubectl rollout restart deployment -n bookinfo productpage-v1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v2 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v3 sleep 20s kubectl rollout restart deployment -n bookinfo ratings-v1 sleep 20s
- Multicluster setup:
kubectl rollout restart deployment -n bookinfo details-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo ratings-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo productpage-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v2 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v3 --context $REMOTE_CONTEXT2 sleep 20s kubectl rollout restart deployment -n bookinfo ratings-v1 --context $REMOTE_CONTEXT2 sleep 20s
- Single cluster setup:
- Verify that your workloads and new gateways point to the new revision.
- Single cluster setup:
istioctl proxy-status
- Multicluster setup:
istioctl proxy-status --context $REMOTE_CONTEXT1
NAME CLUSTER ... ISTIOD VERSION details-v1-7b6df9d8c8-s6kg5.bookinfo cluster1 ... istiod-1-20-7c8f6fd4c4-m9k9t 1.20.8-patch1-solo istio-eastwestgateway-1-20-bdc4fd65f-ftmz9.gloo-mesh-gateways cluster1 ... istiod-1-20-6495985689-rkwwd 1.20.8-patch1-solo productpage-v1-bb494b7d7-xbtxr.bookinfo cluster1 ... istiod-1-20-7c8f6fd4c4-m9k9t 1.20.8-patch1-solo ratings-v1-55b478cfb6-wv2m5.bookinfo cluster1 ... istiod-1-20-7c8f6fd4c4-m9k9t 1.20.8-patch1-solo reviews-v1-6dfcc9fc7d-7k6qh.bookinfo cluster1 ... istiod-1-20-7c8f6fd4c4-m9k9t 1.20.8-patch1-solo reviews-v2-7dddd799b5-m5n2z.bookinfo cluster1 ... istiod-1-20-7c8f6fd4c4-m9k9t 1.20.8-patch1-solo
- Single cluster setup:
- For any namespace where you run Istio-managed apps, change the label to use the new revision. For example, you might update the
To uninstall the previous installations, or if you need to uninstall the canary installations, you can edit your Helm values file to remove the revision entries from the
istioInstallations.controlPlane.installations
andistioInstallations.northSouthGateways.installations
lists. Then, upgrade your Gloo Mesh Helm release with your updated values file.
Testing only: Manually replacing the lifecycle manager CRs
Istio 1.21 is not supported in Gloo Mesh Gateway version 2.5.
Istio 1.20 is supported only as patch version 1.20.1-patch1
and later. Do not use patch versions 1.20.0 and 1.20.1, which contain bugs that impact several Gloo Mesh Gateway features that rely on Istio ServiceEntries.
If you have multiple external services that use the same host and plan to use Istio 1.20, you must use patch version 1.20.7 or later to ensure that the Istio service entry that is created for those external services is correct.
If you manage Istio through your main Gloo Platform Helm chart in testing or demo setups, you can quickly upgrade your Istio service mesh and gateway configurations by manually deleting the IstioLifecycleManager
and GatewayLifecycleManager
CRs, and upgrading your Gloo Mesh installation with your updated gateway values in your Helm values file. Note that you can also use this method to clear your managed Istio configurations if a canary upgrade becomes stuck.
This method is supported only for testing scenarios, because your istiod
control plane and gateways are temporarily removed in this process.
Get the name of your
IstioLifecycleManager
resource. Typically, this resource is namedgloo-platform
.kubectl get IstioLifecycleManager -A --context $MGMT_CONTEXT
Delete the resource.
kubectl delete IstioLifecycleManager gloo-platform -n gloo-mesh --context $MGMT_CONTEXT
Verify that your
istiod
control plane is removed.kubectl get all -n istio-system --context $REMOTE_CONTEXT1 kubectl get all -n istio-system --context $REMOTE_CONTEXT2
Optional: If you also need to make changes to your gateways, clear those configurations.
- Get the name of your
GatewayLifecycleManager
resource. Typically, this resource is namedistio-eastwestgateway
. You might also have anistio-ingressgateway
resource, such as if you use Gloo Mesh Gateway.kubectl get GatewayLifecycleManager -A --context $MGMT_CONTEXT
- Delete the resource.
kubectl delete GatewayLifecycleManager istio-eastwestgateway -n gloo-mesh --context $MGMT_CONTEXT
kubectl delete GatewayLifecycleManager istio-ingressgateway -n gloo-mesh --context $MGMT_CONTEXT
- Verify that your gateway proxy is removed.
kubectl get svc -n gloo-mesh-gateways --context $REMOTE_CONTEXT1 kubectl get svc -n gloo-mesh-gateways --context $REMOTE_CONTEXT2
- Get the name of your
Follow the steps in this guide to perform a regular upgrade of your Gloo Mesh installation and include your Istio changes in your Helm values file. After you apply the updates in your Helm upgrade of the
gloo-platform
chart, Gloo re-creates theistiod
control plane, and if applicable, the Istio gateways.After your Helm upgrade completes, verify that your Istio resources are re-created.
# Change the revision as needed kubectl get all -n gm-iop-1-20 --context $REMOTE_CONTEXT1 kubectl get all -n istio-system --context $REMOTE_CONTEXT1 kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
# Change the revision as needed kubectl get all -n gm-iop-1-20 --context $REMOTE_CONTEXT2 kubectl get all -n istio-system --context $REMOTE_CONTEXT2 kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT2
Upgrade gateways with lifecycle manager CRs
If you manually created IstioLifecycleManager
and GatewayLifecycleManager
CRs to deploy gateway proxies, you can apply updates to your gateway proxies by using an in-place upgrade or a revisioned canary upgrade.
In-place upgrades
Istio 1.21 is not supported in Gloo Mesh Gateway version 2.5.
Istio 1.20 is supported only as patch version 1.20.1-patch1
and later. Do not use patch versions 1.20.0 and 1.20.1, which contain bugs that impact several Gloo Mesh Gateway features that rely on Istio ServiceEntries.
If you have multiple external services that use the same host and plan to use Istio 1.20, you must use patch version 1.20.7 or later to ensure that the Istio service entry that is created for those external services is correct.
In an in-place upgrade, Gloo upgrades your existing control plane or gateway installations. In-place upgrades behave differently based on whether your installation is revisionless or revisioned.
Revisionless (testing only): If your testing-only installation is revisionless (your settings omit the revision
and gatewayRevision
fields), in-place upgrades are triggered when you apply changes to any field in the istioInstallations
section.
Revisioned: If your installation is revisioned, in-place upgrades are triggered only when you apply changes to one of the following fields in the istioInstallations
section. Otherwise, a canary upgrade is required. You can change this behavior by setting the spec.installations.skipUpgradeValidation
boolean value to true
in the IstioLifecycleManager
and GatewayLifecycleManager
resources.
istioOperatorSpec.tag
patch version- In-place upgrades are not supported for downgrading the patch version.
- In-place upgrades are not supported if you do not already specify a
tag
value, such as if you want to switch from theauto
setting to a specific version. This is because you must also specifyhub
andrevision
values, which require a canary upgrade.
istioOperatorSpec.meshConfig
values
Get the names of your managed Istio installation resources.
kubectl get IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT kubectl get GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT
Upgrade your istiod control plane by editing the
IstioLifecycleManager
resource. For example, you might update the patch version of Istio by changing the value ofistioOperatorSpec.tag
. After you save and close the editor, Gloo starts an in-place upgrade of theistiod
control planes.kubectl edit IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <istiod-installation-name>
If you created
GatewayLifecycleManager
resources for ingress, egress, or east-west gateways, you can upgrade your gateways by editing each resource. For example, if you updated the patch version of the control plane, you can also update your gateway patch versions by changing the value ofistioOperatorSpec.tag
. After you save and close the editor, Gloo starts an in-place upgrade of the gateways.If you are updating the Istio image, be sure to update the
istiod
control plane via the IstioLifecycleManager first, before you update your gateways. If you update the gateways before the control plane, the gateways might have an outdated image.kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <ingress-gateway-installation-name>
kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <egress-gateway-installation-name>
Required for versions earlier than 2.5.9, 2.4.16, and 2.3.24: After your Helm upgrade completes, restart your gateway pods in each workload cluster. For example, you might use the following command to rollout a restart of the
istio-ingressgateway-1-20
andistio-egressgateway-1-20
deployments.kubectl rollout restart -n gloo-mesh-gateways deployment/istio-ingressgateway-1-20 --context $REMOTE_CONTEXT1 kubectl rollout restart -n gloo-mesh-gateways deployment/istio-egressgateway-1-20 --context $REMOTE_CONTEXT1
kubectl rollout restart -n gloo-mesh-gateways deployment/istio-ingressgateway-1-20 --context $REMOTE_CONTEXT2 kubectl rollout restart -n gloo-mesh-gateways deployment/istio-egressgateway-1-20 --context $REMOTE_CONTEXT2
Verify that your Istio resources are updated.
kubectl get all -n gm-iop-1-20 --context $REMOTE_CONTEXT1 kubectl get all -n istio-system --context $REMOTE_CONTEXT1 kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
Revisioned canary upgrades
Istio 1.21 is not supported in Gloo Mesh Gateway version 2.5.
Istio 1.20 is supported only as patch version 1.20.1-patch1
and later. Do not use patch versions 1.20.0 and 1.20.1, which contain bugs that impact several Gloo Mesh Gateway features that rely on Istio ServiceEntries.
If you have multiple external services that use the same host and plan to use Istio 1.20, you must use patch version 1.20.7 or later to ensure that the Istio service entry that is created for those external services is correct.
In a canary upgrade, you install another Istio installation (canary) alongside your active installation. Each installation is revisioned so that you can easily identify and verify the separate settings and resources for each installation.
Perform a canary upgrade when you change one of the following fields:
istioOperatorSpec.tag
minor versionistioOperatorSpec.hub
repository, such as switching to the repository for the minor version of the Solo distribution of Istio that you want to upgrade tocomponents
,profile
,values
, ornamespace
in theistioOperatorSpec
Keep in mind the following considerations:
- During a canary upgrade, the validating admissions webhook is enabled only for the canary installation to prevent issues that occur when multiple webhooks are enabled.
- If you plan to run multiple revisions of Istio in your cluster and use
discoverySelectors
in each revision to discover the resources in specific namespaces, enable theglooMgmtServer.extraEnvs.IGNORE_REVISIONS_FOR_VIRTUAL_DESTINATION_TRANSLATION
environment variable on the Gloo management server. For more information, see Multiple Istio revisions in the same cluster.
OpenShift only: Elevate the permissions of the service account that will be created for the new revision’s operator project. This permission allows the Istio sidecar to make use of a user ID that is normally restricted by OpenShift. Replace the revision with the revision you plan to use.
oc adm policy add-scc-to-group anyuid system:serviceaccounts:gm-iop-1-20 --context $REMOTE_CONTEXT1 oc adm policy add-scc-to-group anyuid system:serviceaccounts:gm-iop-1-20 --context $REMOTE_CONTEXT2
Get the names of your managed Istio installation resources.
kubectl get IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT kubectl get GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT
Edit the
IstioLifecycleManager
resource for theistiod
control plane to add another installation entry for the canary revision. For the canary revision, be sure to setdefaultRevision
tofalse
so that only the existing revision continues to run.kubectl edit IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <installation-name>
For example, if you upgrade to Istio 1.20.8-patch1, add the following
1-20
canary revision as a new entry in theinstallations
section:apiVersion: admin.gloo.solo.io/v2 kind: IstioLifecycleManager metadata: name: istiod-control-plane namespace: gloo-mesh spec: installations: # Existing revision - revision: 1-19 clusters: - name: cluster1 # Keep this field set to TRUE so that only the existing revision continues to run defaultRevision: true - name: cluster2 defaultRevision: true istioOperatorSpec: ... # Canary revision - revision: 1-20 clusters: - name: cluster1 # Set this field to FALSE so that only the existing revision continues to run defaultRevision: false - name: cluster2 defaultRevision: false istioOperatorSpec: ...
Updating the minor version of Istio? In your canary revision section, be sure to update both the repo key in the
hub
field, and the Istio version in thetag
field. You can get the repo key for the Istio version that you want to install from the Istio images built by Solo.io support article.If you created
GatewayLifecycleManager
resources for ingress, egress, or east-west gateways, edit those resources to add another installation entry for the canary revision. Be sure to setactiveGateway
tofalse
so that only the existing revision continues to run.If you are updating the Istio image, be sure to update the
istiod
control plane via the IstioLifecycleManager first, before you update your gateways. If you update the gateways before the control plane, the gateways might have an outdated image.
If you disable the load balancer services that are defined by default in the GatewayLifecycleManager, and instead maintain your own services to expose the gateways, you can later switch from the old to the new gateways by updating the services to point to the new gateways. However, note that due to an upstream Istio bug, it can takeistiod
up to 10 minutes to reconfigure the gateway listener for the updated ports on the service. To avoid this delay, you can expose the canary gateway with at least a ClusterIP service so that the control plane detects the service’s ports immediately at deployment time.kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <installation-name>
For example, if you upgrade to Istio 1.20.8-patch1, add the following
1-20
canary revision as a new entry in theinstallations
section:apiVersion: admin.gloo.solo.io/v2 kind: GatewayLifecycleManager metadata: name: istio-ingressgateway namespace: gloo-mesh spec: installations: # Existing revision - gatewayRevision: 1-19 clusters: - name: cluster1 # Keep this field set to TRUE so that only the existing revision continues to run activeGateway: true - name: cluster2 activeGateway: true istioOperatorSpec: profile: empty ... # Canary revision - gatewayRevision: 1-20 clusters: - name: cluster1 # Set this field to FALSE so that only the existing revision continues to run activeGateway: false - name: cluster2 activeGateway: false istioOperatorSpec: profile: empty ...
For most use cases, you can set the
revision
for the IstioLifecycleManager and thegatewayRevision
for the GatewayLifecycleManager to the same version. However, gateway installations can point to any istiod control plane revision by using thecontrolPlaneRevision
field. For simplicity, if you do not specifycontrolPlaneRevision
, the gateway installation uses a control plane with the same revision as itself.Verify that Istio resources for the canary installation are created. For example, if you updated the Istio minor version to
1.20.8-patch1
, verify that resources are created in thegm-iop-1-20
namespace, and that resources for1-20
are created alongside the existing resources for the previous version in theistio-system
andgloo-mesh-gateways
namespaces. Note that the gateway load balancers for the canary revision contain the revision in the name, such asistio-ingressgateway-1-20
.kubectl get all -n gm-iop-1-20 --context $REMOTE_CONTEXT1 kubectl get all -n istio-system --context $REMOTE_CONTEXT1 kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
kubectl get all -n gm-iop-1-20 --context $REMOTE_CONTEXT2 kubectl get all -n istio-system --context $REMOTE_CONTEXT2 kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT2
After performing any necessary testing, switch to the new
istiod
control plane revision by changingdefaultRevision
tofalse
for the old revision and totrue
for the new revision.kubectl edit IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT istiod-control-plane
Example:
apiVersion: admin.gloo.solo.io/v2 kind: IstioLifecycleManager metadata: name: istiod-control-plane namespace: gloo-mesh spec: installations: # Old revision - revision: 1-19 clusters: - name: cluster1 # Set this field to FALSE defaultRevision: false - name: cluster2 defaultRevision: false istioOperatorSpec: ... # New revision - revision: 1-20 clusters: - name: cluster1 # Set this field to TRUE defaultRevision: true - name: cluster2 defaultRevision: true istioOperatorSpec: ...
In your workload clusters, update the labels on your service namespaces to use the new revision.
For any namespace where you run Istio-managed apps, change the label to use the new revision, such as in these example commands.
kubectl label ns <namespace> --overwrite istio.io/rev=1-20 --context $REMOTE_CONTEXT1 kubectl label ns <namespace> --overwrite istio.io/rev=1-20 --context $REMOTE_CONTEXT2
For example, if you installed the Bookinfo sample app:
kubectl label ns bookinfo --overwrite istio.io/rev=1-20 --context $REMOTE_CONTEXT1 kubectl label ns bookinfo --overwrite istio.io/rev=1-20 --context $REMOTE_CONTEXT2
If you did not previously use revision labels for your apps, you can upgrade your application's sidecars by running <code>kubectl label ns <namespace> istio-injection-</code> and <code>kubectl label ns <namespace> istio.io/rev=<revision></code>.
Update any Istio-managed apps by rolling out restarts. The Istio sidecars for each microservice are updated to use the new Istio version. Make sure that you only restart one microservice at a time. For example, in the following commands to update the Bookinfo microservices, 20 seconds elapse between each restart to ensure that the pods have time to start running.
kubectl rollout restart deployment -n bookinfo details-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo ratings-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo productpage-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v2 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v3 --context $REMOTE_CONTEXT2 sleep 20s kubectl rollout restart deployment -n bookinfo ratings-v1 --context $REMOTE_CONTEXT2 sleep 20s
Verify that your workloads and new gateways point to the new revision.
istioctl proxy-status --context $REMOTE_CONTEXT1
Example output:
NAME CLUSTER ... ISTIOD VERSION details-v1-7b6df9d8c8-s6kg5.bookinfo cluster1 ... istiod-1-20-7c8f6fd4c4-m9k9t 1.20.8-patch1-solo istio-ingressgateway-1-20-bdc4fd65f-ftmz9.gloo-mesh-gateways cluster1 ... istiod-1-20-6495985689-rkwwd 1.20.8-patch1-solo productpage-v1-bb494b7d7-xbtxr.bookinfo cluster1 ... istiod-1-20-7c8f6fd4c4-m9k9t 1.20.8-patch1-solo ratings-v1-55b478cfb6-wv2m5.bookinfo cluster1 ... istiod-1-20-7c8f6fd4c4-m9k9t 1.20.8-patch1-solo reviews-v1-6dfcc9fc7d-7k6qh.bookinfo cluster1 ... istiod-1-20-7c8f6fd4c4-m9k9t 1.20.8-patch1-solo reviews-v2-7dddd799b5-m5n2z.bookinfo cluster1 ... istiod-1-20-7c8f6fd4c4-m9k9t 1.20.8-patch1-solo
Switch to any new gateways by changing
activeGateway
tofalse
for the old revision and totrue
for the new revision.kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <installation-name>
Example:
apiVersion: admin.gloo.solo.io/v2 kind: GatewayLifecycleManager metadata: name: istio-ingressgateway namespace: gloo-mesh spec: installations: # Old revision - gatewayRevision: 1-19 clusters: - name: cluster1 # Set this field set to FALSE activeGateway: false - name: cluster2 activeGateway: false istioOperatorSpec: profile: empty ... # New revision - gatewayRevision: 1-20 clusters: - name: cluster1 # Set this field to TRUE activeGateway: true - name: cluster2 activeGateway: true istioOperatorSpec: profile: empty ...
New load balancers are created for the canary gateways. To instead change the control plane revision in use by the existing gateway load balancers, you can set the
istio.io/rev
label on the gateway deployment, which triggers a rolling restart.Verify that the active gateways for the new revision are created, which do not have the revision appended to the name. Note that gateways for the inactive revision that you previously ran also exist in the namespace, in the case that a rollback is required.
kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1 kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT2
Example output, in which the active gateway (
istio-ingressgateway
) for the new revision and inactive gateway (such asistio-ingressgateway-1-19
) for the old revision are created:NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE istio-ingressgateway LoadBalancer 10.44.4.140 34.150.235.221 15021:31321/TCP,80:32525/TCP,443:31826/TCP 48s istio-ingressgateway-1-19 LoadBalancer 10.56.15.36 34.145.163.61 15021:31936/TCP,80:30196/TCP,443:32286/TCP,15443:31851/TCP 45s
To uninstall the previous installation, or if you need to uninstall the canary installation, you can edit the
IstioLifecycleManager
andGatewayLifecycleManager
resources to remove the revision’s entry from theinstallations
list.