Upgrade Gloo Mesh-managed Istio
Use Gloo Mesh to upgrade your Gloo Mesh-managed Istio instalations in your workload clusters.
This guide upgrades Istio installations that are managed directly by the IstioLifecycleManager
and GatewayLifecycleManager
CRDs you created. If you use the istioInstallations
section of your Gloo Mesh Helm chart to manage the Istio control planes and gateways instead of using CRDs, follow the Upgrade Helm-managed Istio guide instead.
About
After you use the lifecycle manager CRDs to deploy Istio installations, changes that you make to the IstioLifecycleManager
and GatewayLifecycleManager
resources are automatically propagated to your Istio installations. In some cases, these changes might require updates to your control plane or gateway deployments. Depending on the type of change, you apply updates to the installations in one of the following ways:
- Canary upgrade (recommended)
- In-place upgrade
- Testing only: Manually replacing the IstioLifecycleManager CR
Canary upgrades (recommended)
In a canary upgrade, you install another Istio installation (canary) alongside your active installation. Note that during a canary upgrade, the validating admissions webhook is enabled only for the canary installation to prevent issues that occur when multiple webhooks are enabled. Note that you can force Gloo to always use canary upgrades by setting the spec.upgradeStrategy.alwaysUseCanaryUpgrade
boolean value to true
in the IstioLifecycleManager
and GatewayLifecycleManager
resources.
Perform a canary upgrade when you change one of the following fields:
- Minor version in the
tag
field of theistioOperatorSpec
hub
value in theistioOperatorSpec
, such as switching from the default Istio build to a Solo buildcomponents
,profile
,values
, ornamespace
in theistioOperatorSpec
In-place upgrades
In an in-place upgrade, Gloo upgrades your existing control plane or gateway installations. In-place upgrades are triggered when you change one of the following fields:
- Patch version in the
tag
field of theistioOperatorSpec
- In-place upgrades are not supported for downgrading the patch version.
- In-place upgrades are not supported if you do not already specify a
tag
value, such as if you want to switch from theauto
setting to a specific version. This is because you must also specifyhub
andrevision
values, which require a canary upgrade.
meshConfig
values in theistioOperatorSpec
Testing only: Manually replacing the GatewayLifecycleManager CR
If you manage Istio through your main Gloo Mesh Helm chart in testing or demo setups, you can quickly upgrade your Istio service mesh and gateway configurations by manually deleting the IstioLifecycleManager
and GatewayLifecycleManager
CRs, and upgrading your Gloo Mesh installation with your updated gateway values in your Helm values file. This method is supported only for testing scenarios, because your istiod
control plane and gateways are temporarily removed in this process. You can also use this method to clear your managed Istio configurations if a canary upgrade becomes stuck.
Performing a canary upgrade
Update the minor version of Istio, the repository of the Istio image, or the components
, profile
, values
, or namespace
of the Istio installations.
Istio version 1.17 does not support the Gloo legacy metrics pipeline. If you run the legacy metrics pipeline, before you upgrade or install Istio with version 1.17, be sure that you set up the Gloo OpenTelemetry (OTel) pipeline instead in your new or existing Gloo Mesh installation.
-
OpenShift only: Elevate the permissions of the service account that will be created for the new revision's operator project. This permission allows the Istio sidecar to make use of a user ID that is normally restricted by OpenShift. Replace the revision with the revision you plan to use.
oc adm policy add-scc-to-group anyuid system:serviceaccounts:gm-iop-1-17-2 --context $REMOTE_CONTEXT1 oc adm policy add-scc-to-group anyuid system:serviceaccounts:gm-iop-1-17-2 --context $REMOTE_CONTEXT2
-
Get the names of your managed Istio installation resources.
kubectl get IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT kubectl get GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT
-
Edit the
IstioLifecycleManager
resource for theistiod
control plane to add another installation entry for the canary revision. For the canary revision, be sure to setdefaultRevision
tofalse
so that only the existing revision continues to run.kubectl edit IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <installation-name>
For example, if you upgrade to Istio 1.17, add the following
1-17-2
canary revision as a new entry in theinstallations
section:apiVersion: admin.gloo.solo.io/v2 kind: IstioLifecycleManager metadata: name: istiod-control-plane namespace: gloo-mesh spec: installations: # Existing revision - revision: 1-16-4 clusters: - name: cluster1 # Keep this field set to TRUE so that only the existing revision continues to run defaultRevision: true - name: cluster2 defaultRevision: true istioOperatorSpec: profile: minimal ... # Canary revision - revision: 1-17-2 clusters: - name: cluster1 # Set this field to FALSE so that only the existing revision continues to run defaultRevision: false - name: cluster2 defaultRevision: false istioOperatorSpec: profile: minimal ...
Updating the minor version of Istio? In your canary revision section, be sure to update both the repo key in the
hub
field, and the Istio version in thetag
field. You can get the repo key for the Istio version that you want to install from the Istio images built by Solo.io support article. -
If you created
GatewayLifecycleManager
resources for east-west or ingress gateways, edit those resources to add another installation entry for the canary revision. Be sure to setactiveGateway
tofalse
so that only the existing revision continues to run.kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <installation-name>
For example, if you upgrade to Istio 1.17, add the following
1-17-2
canary revision as a new entry in theinstallations
section:apiVersion: admin.gloo.solo.io/v2 kind: GatewayLifecycleManager metadata: name: istio-ingressgateway namespace: gloo-mesh spec: installations: # Existing revision - gatewayRevision: 1-16-4 clusters: - name: cluster1 # Keep this field set to TRUE so that only the existing revision continues to run activeGateway: true - name: cluster2 activeGateway: true istioOperatorSpec: profile: empty ... # Canary revision - gatewayRevision: 1-17-2 clusters: - name: cluster1 # Set this field to FALSE so that only the existing revision continues to run activeGateway: false - name: cluster2 activeGateway: false istioOperatorSpec: profile: empty ...
For most use cases, you can set the
revision
for theIstioLifecycleManager
and thegatewayRevision
for theGatewayLifecycleManager
to the same version. However, gateway installations can point to anyistiod
control plane revision by using thecontrolPlaneRevision
field. For simplicity, if you do not specifycontrolPlaneRevision
, the gateway installation uses a control plane with the same revision as itself. -
Verify that Istio resources for the canary installation are created. For example, if you updated the Istio minor version to
1-17-2
, verify that resources are created in thegm-iop-1-17-2
namespace, and that resources for1-17-2
are created alongside the existing resources for the previous version in theistio-system
andgloo-mesh-gateways
namespaces. Note that the gateway load balancers for the canary revision contain the revision in the name, such asistio-ingressgateway-1-17-2
.kubectl get all -n gm-iop-1-17-2 kubectl get all -n istio-system kubectl get all -n gloo-mesh-gateways
kubectl get all -n gm-iop-1-17-2 --context $REMOTE_CONTEXT1 kubectl get all -n istio-system --context $REMOTE_CONTEXT1 kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
kubectl get all -n gm-iop-1-17-2 --context $REMOTE_CONTEXT2 kubectl get all -n istio-system --context $REMOTE_CONTEXT2 kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT2
Running into issues or seeing a stuck canary upgrade? In testing environments, you can clear your configuration by manually replacing the
GatewayLifecycleManager
CR. -
After performing any necessary testing, switch to the new
istiod
control plane revision by changingdefaultRevision
tofalse
for the old revision and totrue
for the new revision.kubectl edit IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT istiod-control-plane
Example:
apiVersion: admin.gloo.solo.io/v2 kind: IstioLifecycleManager metadata: name: istiod-control-plane namespace: gloo-mesh spec: installations: # Old revision - revision: 1-16-4 clusters: - name: cluster1 # Set this field to FALSE defaultRevision: false - name: cluster2 defaultRevision: false istioOperatorSpec: profile: minimal ... # New revision - revision: 1-17-2 clusters: - name: cluster1 # Set this field to TRUE defaultRevision: true - name: cluster2 defaultRevision: true istioOperatorSpec: profile: minimal ...
-
Upgrade your apps’ Istio sidecars.
- For any namespace where you run Istio-managed apps, change the label to use the new revision. For example, you might update the
bookinfo
namespace.If you did not previously use revision labels for your apps, you can upgrade your application's sidecars by running ‘kubectl label ns bookinfo istio-injection-’ and ‘kubectl label ns bookinfo istio.io/rev=$REVISION’.kubectl label ns bookinfo istio.io/rev=$REVISION --overwrite
kubectl label ns bookinfo istio.io/rev=$REVISION --overwrite --context $REMOTE_CONTEXT1 kubectl label ns bookinfo istio.io/rev=$REVISION --overwrite --context $REMOTE_CONTEXT2
- Update any Istio-managed apps by rolling out restarts. The Istio sidecars for each microservice are updated to use the new Istio version. Make sure that you only restart one microservice at a time. For example, in the following commands to update the Bookinfo microservices, 20 seconds elapse between each restart to ensure that the pods have time to start running.
kubectl rollout restart deployment -n bookinfo details-v1 sleep 20s kubectl rollout restart deployment -n bookinfo productpage-v1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v2 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v3 sleep 20s kubectl rollout restart deployment -n bookinfo ratings-v1 sleep 20s
kubectl rollout restart deployment -n bookinfo details-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo ratings-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo productpage-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v1 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v2 --context $REMOTE_CONTEXT1 sleep 20s kubectl rollout restart deployment -n bookinfo reviews-v3 --context $REMOTE_CONTEXT2 sleep 20s kubectl rollout restart deployment -n bookinfo ratings-v1 --context $REMOTE_CONTEXT2 sleep 20s
- Verify that your workloads and new gateways point to the new revision.
istioctl proxy-status
istioctl proxy-status --context $REMOTE_CONTEXT1
NAME CLUSTER ... ISTIOD VERSION details-v1-7b6df9d8c8-s6kg5.bookinfo cluster1 ... istiod-1-17-2-7c8f6fd4c4-m9k9t 1.17.2-solo istio-eastwestgateway-1-17-2-bdc4fd65f-ftmz9.gloo-mesh-gateways cluster1 ... istiod-1-17-2-6495985689-rkwwd 1.17.2-solo productpage-v1-bb494b7d7-xbtxr.bookinfo cluster1 ... istiod-1-17-2-7c8f6fd4c4-m9k9t 1.17.2-solo ratings-v1-55b478cfb6-wv2m5.bookinfo cluster1 ... istiod-1-17-2-7c8f6fd4c4-m9k9t 1.17.2-solo reviews-v1-6dfcc9fc7d-7k6qh.bookinfo cluster1 ... istiod-1-17-2-7c8f6fd4c4-m9k9t 1.17.2-solo reviews-v2-7dddd799b5-m5n2z.bookinfo cluster1 ... istiod-1-17-2-7c8f6fd4c4-m9k9t 1.17.2-solo
- For any namespace where you run Istio-managed apps, change the label to use the new revision. For example, you might update the
-
Switch to any new gateways by changing
activeGateway
tofalse
for the old revision and totrue
for the new revision.kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <installation-name>
Example:
apiVersion: admin.gloo.solo.io/v2 kind: GatewayLifecycleManager metadata: name: istio-ingressgateway namespace: gloo-mesh spec: installations: # Old revision - gatewayRevision: 1-16-4 clusters: - name: cluster1 # Keep this field set to FALSE activeGateway: false - name: cluster2 activeGateway: false istioOperatorSpec: profile: empty ... # New revision - gatewayRevision: 1-17-2 clusters: - name: cluster1 # Set this field to TRUE activeGateway: true - name: cluster2 activeGateway: ture istioOperatorSpec: profile: empty ...
New load balancers are created for the canary gateways. To instead change the control plane revision in use by the existing gateway load balancers, you can set the
istio.io/rev
label on the gateway deployment, which triggers a rolling restart. -
Verify that the active gateways for the new revision are created, which do not have the revision appended to the name. Note that gateways for the inactive revision that you previously ran also exist in the namespace, in the case that a rollback is required.
kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1 kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT2
Example output, in which the active gateway (
istio-ingressgateway
) for the new revision and inactive gateway (such asistio-ingressgateway-1-16-4
) for the old revision are created:NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) istio-ingressgateway LoadBalancer 10.44.4.140 34.150.235.221 15021:31321/TCP,80:32525/TCP,443:31826/TCP 48s AGE istio-ingressgateway-1-16-4 LoadBalancer 10.56.15.36 34.145.163.61 15021:31936/TCP,80:30196/TCP,443:32286/TCP,15443:31851/TCP 45s
-
To uninstall the previous installation, or if you need to uninstall the canary installation, you can edit the
IstioLifecycleManager
andGatewayLifecycleManager
resources to remove the revision's entry from theinstallations
list.
Performing an in-place upgrade
Update the patch version of Istio or update meshConfig values of the Istio installations.
Istio version 1.17 does not support the Gloo legacy metrics pipeline. If you run the legacy metrics pipeline, before you upgrade or install Istio with version 1.17, be sure that you set up the Gloo OpenTelemetry (OTel) pipeline instead in your new or existing Gloo Mesh installation.
-
Get the names of your managed Istio installation resources.
kubectl get IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT kubectl get GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT
-
Upgrade your istiod control plane by editing the
IstioLifecycleManager
resource. For example, you might update the patch version of Istio by changing the value ofistioOperatorSpec.tag
. After you save and close the editor, Gloo starts an in-place upgrade of theistiod
control planes.kubectl edit IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <istiod-installation-name>
-
If you created
GatewayLifecycleManager
resources for ingress or east-west gateways, you can upgrade your gateways by editing each resource. For example, if you updated the patch version of the control plane, you can also update your gateway patch versions by changing the value ofistioOperatorSpec.tag
. After you save and close the editor, Gloo starts an in-place upgrade of the gateways.kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <ingress-gateway-installation-name>
kubectl edit GatewayLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <eastwest-gateway-installation-name>
-
After your Helm upgrade completes, restart your gateway pods in each workload cluster. For example, you might use the following command to rollout a restart of the
istio-eatwestgateway-1-17-2
andistio-ingressgateway-1-17-2
deployments.kubectl rollout restart -n gloo-mesh-gateways deployment/istio-eastwestgateway-1-17-2 --context $REMOTE_CONTEXT1 kubectl rollout restart -n gloo-mesh-gateways deployment/istio-ingressgateway-1-17-2 --context $REMOTE_CONTEXT1
kubectl rollout restart -n gloo-mesh-gateways deployment/istio-eastwestgateway-1-17-2 --context $REMOTE_CONTEXT2 kubectl rollout restart -n gloo-mesh-gateways deployment/istio-ingressgateway-1-17-2 --context $REMOTE_CONTEXT2
-
Verify that your Istio resources are updated.
kubectl get all -n gm-iop-1-17-2 --context $REMOTE_CONTEXT1 kubectl get all -n istio-system --context $REMOTE_CONTEXT1 kubectl get all -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
Testing only: Manually replacing the GatewayLifecycleManager CR
This method is supported only for testing scenarios, because your istiod
control plane and gateways are temporarily removed in this process.
-
Get the name of your
IstioLifecycleManager
resource. Typically, this resource is namedgloo-platform
.kubectl get IstioLifecycleManager -A
-
Delete the resource.
kubectl delete IstioLifecycleManager gloo-platform -n gloo-mesh
-
Verify that your
istiod
control plane is removed.kubectl get all -n istio-system
-
Optional: If you also need to make changes to your gateways, clear those configurations.
- Get the name of your
GatewayLifecycleManager
resource. Typically, this resource is namedistio-eastwestgateway
. You might also have anistio-ingressgateway
resource, such as if you use Gloo Gateway.kubectl get GatewayLifecycleManager -A
- Delete the resource.
kubectl delete GatewayLifecycleManager istio-eastwestgateway -n gloo-mesh
kubectl delete GatewayLifecycleManager istio-ingressgateway -n gloo-mesh
- Verify that your gateway proxy is removed.
kubectl get svc -n gloo-mesh-gateways
- Get the name of your
-
Follow the steps in the upgrade guide to perform a regular upgrade of your Gloo Mesh installation and include your Istio changes in your Helm values file. After you apply the updates in your Helm upgrade of the
gloo-platform
chart, Gloo re-creates theistiod
control plane, and if applicable, the Istio gateways. -
After your Helm upgrade completes, verify that your Istio resources are re-created.
# Change the revision as needed kubectl get all -n gm-iop-1-17-2 kubectl get all -n istio-system kubectl get all -n gloo-mesh-gateways