On this page

Upgrade ambient meshes with Helm

Upgrade your ambient control and data plane components in your Gloo Mesh cluster.

Considerations

Before you upgrade your ambient components, review the following limitations and recommendations.

Revision and canary upgrade limitations

The upgrade guides in this documentation show you how to perform in-place upgrades for your Istio components, which is the recommended upgrade strategy.

CNI upgrades

In an ambient installation, the istio-cni pods are enhanced for safe in-place upgrades. When istio-cni is scheduled for removal, it determines whether an upgrade is in process by checking the value of AGENT_UPGRADE_ID in the istio-cni ConfigMap against its internal upgrade ID. If the removal is due to an upgrade, istio-cni creates a “stub” CNI plug-in in its place on the node. This stub CNI plug-in stalls any newly scheduled pods on that node, and does not allow the pods to spin up until the replacement istio-cni is up and running.

ztunnel upgrades

As Kubernetes dynamically upgrades the ztunnel deployment by restarting the old ztunnel pods, all long-lived connections on the node are dropped. You can influence the duration of the downtime with the termination grace period of the ztunnel pod, which determines the drain period. Any TCP connections that are longer than the grace period are dropped. Alternatively, you might cordon nodes and use blue/green node pools to limit the number of active pods that are in the process of upgrading.

Version and license requirements

Verify that the minor version of the Solo distribution of Istio that you want to upgrade to is tested and supported for your Gloo Mesh version. To find the available patch versions, you can get the minor version repo URL from the Istio images built by Solo.io support article, and check the patch version builds in that repo.
Check the Istio release notes for the upgrade version to prepare for any breaking changes.
Be sure to review the following known Istio version restrictions.
- Single cluster setups:
  - In Gloo Mesh version 2.6 and later, ambient mode requires the Solo distribution of Istio version 1.22.3 or later (1.22.3-solo).
  - In Istio 1.22.0-1.22.3, the ISTIO_DELTA_XDS environment variable must be set to false. For more information, see this upstream Istio issue. Note that this issue is resolved in Istio 1.22.4.
- Multicluster setups:
  - In Gloo Mesh version 2.7 and later, multicluster setups require the Solo distribution of Istio version 1.24.3 or later (1.24.3-solo), including the Solo distribution of istioctl.
  - This feature requires your mesh to be installed with the Solo distribution of Istio and an Enterprise-level license for Gloo Mesh. Contact your account representative to obtain a valid license.

Single cluster

Upgrade istioctl

Save the details for the version of the Solo distribution of Istio that you want to upgrade to.

  export ISTIO_VERSION=1.25.2
# Change the tags as needed
export ISTIO_IMAGE=${ISTIO_VERSION}-solo
export REPO_KEY=e038d180f90a
export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}

Save the Solo distribution of Istio patch version and tag.

  export ISTIO_VERSION=1.24.5
# Change the tags as needed
export ISTIO_IMAGE=${ISTIO_VERSION}-solo

Save the repo key for the minor version of the Solo distribution of Istio that you want to install. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Ambient section of the Istio images built by Solo.io support article.
```
  # 12-character hash at the end of the minor version repo URL
export REPO_KEY=<repo_key>
export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}
  
```

Upgrade your istioctl CLI client to the new version.

  curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -
cd istio-${ISTIO_VERSION}
export PATH=$PWD/bin:$PATH

Upgrade CRDs and istiod

Upgrade the custom resources of the Kubernetes Gateway API to the latest supported version, 1.2.1.

  kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

Upgrade the Istio CRDs to the new version.

  helm get values istio-base -n istio-system -o yaml > istio-base.yaml
helm upgrade istio-base oci://${HELM_REPO}/base \
--namespace istio-system \
--version ${ISTIO_IMAGE} \
-f istio-base.yaml

notifications

If you see an error such as Error: UPGRADE FAILED: Rendered manifests contain a resource that already exists, see the community Istio docs.

Get the current values for the istiod Helm release in your cluster.

  helm get values istiod -n istio-system -o yaml > istiod.yaml
open istiod.yaml

Make edits to the istiod Helm values, and save the file. If you update the Istio minor version, such as in the global.tag field, be sure to also update the value of the hub field to the repo for the correct version of the Solo distribution of Istio.

Upgrade your Helm release with the updated values.

  helm upgrade istiod oci://${HELM_REPO}/istiod \
-n istio-system \
--version ${ISTIO_IMAGE} \
-f istiod.yaml

Verify that the istiod pods are successfully restarted. Note that it might take a few seconds for the pods to become available.
```
  kubectl get pods -n istio-system | grep istiod
  
```
Example output:
```
  istiod-b84c55cff-tllfr   1/1     Running   0          58s
  
```

Upgrade the CNI and ztunnel

info

If your changes include upgrading the Istio version of the components, be sure to upgrade the istiod control plane before you upgrade the CNI and ztunnel components. Otherwise, these components might have an outdated image.

For the component that you want to update, get the current values for its Helm release in your cluster.

  helm get values istio-cni -n istio-system -o yaml > cni.yaml
open cni.yaml

  helm get values ztunnel -n istio-system -o yaml > ztunnel.yaml
open ztunnel.yaml

  helm get values istio-cni -n kube-system -o yaml > cni.yaml
open cni.yaml

  helm get values ztunnel -n kube-system -o yaml > ztunnel.yaml
open ztunnel.yaml

Make edits to the Helm values of the components that you want to upgrade, and save the files. If you update the Istio minor version, such as in tag fields, be sure to also update the value of the hub field to the repo for the correct version of the Solo distribution of Istio.

Upgrade your Helm releases with the updated values.

  helm upgrade istio-cni oci://${HELM_REPO}/cni -n istio-system --version ${ISTIO_IMAGE} -f cni.yaml

  helm upgrade ztunnel oci://${HELM_REPO}/ztunnel -n istio-system --version ${ISTIO_IMAGE} -f ztunnel.yaml

  helm upgrade istio-cni oci://${HELM_REPO}/cni -n kube-system --version ${ISTIO_IMAGE} -f cni.yaml

  helm upgrade ztunnel oci://${HELM_REPO}/ztunnel -n kube-system --version ${ISTIO_IMAGE} -f ztunnel.yaml

Verify that the ztunnel and Istio CNI pods are successfully restarted. Note that it might take a few seconds for the pods to become available.

  kubectl get pods -n istio-system

  kubectl get pods -n kube-system

Example output:

  istiod-85c4dfd97f-mncj5            1/1     Running   0             42s
istio-cni-node-pr5rl               1/1     Running   0             42s
istio-cni-node-pvmx2               1/1     Running   0             42s
istio-cni-node-lcrcd               1/1     Running   0             42s
ztunnel-tvtzn                      1/1     Running   0             40s
ztunnel-vtpjm                      1/1     Running   0             40s
ztunnel-hllxg                      1/1     Running   0             40s

Multicluster

Upgrade the multicluster ambient mesh in your multicluster setup.

Upgrade istioctl

Save the details for the version of the Solo distribution of Istio that you want to upgrade to.

  # Change the tags as needed
export ISTIO_IMAGE=1.25.2-solo
export REPO_KEY=e038d180f90a
export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}

Save the Solo distribution of Istio patch version and tag.

  # Change the tags as needed
export ISTIO_IMAGE=1.24.5-solo

Save the repo key for version 1.24 of the Solo distribution of Istio. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Ambient section of the Istio images built by Solo.io support article.
```
  # 12-character hash at the end of the 1.24 repo URL
export REPO_KEY=<repo_key>
export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}
  
```

Upgrade your istioctl CLI client to the new version.

Get the OS and architecture that you use on your machine.

  OS=$(uname | tr '[:upper:]' '[:lower:]' | sed -E 's/darwin/osx/')
ARCH=$(uname -m | sed -E 's/aarch/arm/; s/x86_64/amd64/; s/armv7l/armv7/')
echo $OS
echo $ARCH

Download the Solo distribution of Istio binary and install istioctl.

  mkdir -p ~/.istioctl/bin
curl -sSL https://storage.googleapis.com/istio-binaries-$REPO_KEY/$ISTIO_IMAGE/istioctl-$ISTIO_IMAGE-$OS-$ARCH.tar.gz | tar xzf - -C ~/.istioctl/bin
chmod +x ~/.istioctl/bin/istioctl

export PATH=${HOME}/.istioctl/bin:${PATH}

Verify that the istioctl client is upgraded.

  istioctl version --remote=false

Example output:

  client version: 1.25.2-solo

Upgrade CRDs and istiod

Save the name and kubeconfig context of a cluster where you want to upgrade Istio in the following environment variables. Each time you repeat the steps in this guide, you change these variables to the next workload cluster’s name and context.
```
  export CLUSTER_NAME=<cluster-name>
export CLUSTER_CONTEXT=<cluster-context>
  
```

Upgrade the custom resources of the Kubernetes Gateway API to the latest supported version, 1.2.1.

  kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

Upgrade the Istio CRDs to the new version.

  helm get values --kube-context ${CLUSTER_CONTEXT} istio-base -n istio-system -o yaml > istio-base.yaml
helm upgrade istio-base oci://${HELM_REPO}/base \
--namespace istio-system \
--kube-context ${CLUSTER_CONTEXT} \
--version ${ISTIO_IMAGE} \
-f istio-base.yaml

notifications

If you see an error such as Error: UPGRADE FAILED: Rendered manifests contain a resource that already exists, see the community Istio docs.

Get the current values for the istiod Helm release in your cluster.

  helm get values --kube-context ${CLUSTER_CONTEXT} istiod -n istio-system -o yaml > istiod.yaml
open istiod.yaml

Make edits to the istiod Helm values, and save the file. If you update the Istio minor version, such as in the global.tag field, be sure to also update the value of the hub field to the repo for the correct version of the Solo distribution of Istio.

Upgrade your Helm release with the updated values.

  helm upgrade istiod oci://${HELM_REPO}/istiod \
-n istio-system \
--version ${ISTIO_IMAGE} \
--kube-context ${CLUSTER_CONTEXT} \
-f istiod.yaml

Verify that the istiod pods are successfully restarted. Note that it might take a few seconds for the pods to become available.

  kubectl get pods --context ${CLUSTER_CONTEXT} -n istio-system | grep istiod

Example output:

  istiod-b84c55cff-tllfr   1/1     Running   0          58s

Upgrade the CNI and ztunnel

info

For the component that you want to update, get the current values for its Helm release in your cluster.

Kubernetes:

  helm get values istio-cni --kube-context ${CLUSTER_CONTEXT} -n istio-system -o yaml > cni.yaml
open cni.yaml

helm get values ztunnel --kube-context ${CLUSTER_CONTEXT} -n istio-system -o yaml > ztunnel.yaml
open ztunnel.yaml

OpenShift:

  helm get values istio-cni --kube-context ${CLUSTER_CONTEXT} -n kube-system -o yaml > cni.yaml
open cni.yaml

helm get values ztunnel --kube-context ${CLUSTER_CONTEXT} -n kube-system -o yaml > ztunnel.yaml
open ztunnel.yaml

Make edits to the Helm values of the components that you want to upgrade, and save the files. If you update the Istio minor version, such as in tag fields, be sure to also update the value of the hub field to the repo for the correct version of the Solo distribution of Istio.

Upgrade your Helm releases with the updated values.

Kubernetes:

  helm upgrade istio-cni oci://${HELM_REPO}/cni --kube-context ${CLUSTER_CONTEXT} -n istio-system --version ${ISTIO_IMAGE} -f cni.yaml

helm upgrade ztunnel oci://${HELM_REPO}/ztunnel --kube-context ${CLUSTER_CONTEXT} -n istio-system --version ${ISTIO_IMAGE} -f ztunnel.yaml

OpenShift:

  helm upgrade istio-cni oci://${HELM_REPO}/cni --kube-context ${CLUSTER_CONTEXT} -n kube-system --version ${ISTIO_IMAGE} -f cni.yaml

helm upgrade ztunnel oci://${HELM_REPO}/ztunnel --kube-context ${CLUSTER_CONTEXT} -n kube-system --version ${ISTIO_IMAGE} -f ztunnel.yaml

Verify that the ztunnel and Istio CNI pods are successfully restarted. Note that it might take a few seconds for the pods to become available.

  kubectl get pods --context ${CLUSTER_CONTEXT} -n istio-system

  kubectl get pods --context ${CLUSTER_CONTEXT} -n kube-system

Example output:

  istiod-85c4dfd97f-mncj5            1/1     Running   0             42s
istio-cni-node-pr5rl               1/1     Running   0             42s
istio-cni-node-pvmx2               1/1     Running   0             42s
istio-cni-node-lcrcd               1/1     Running   0             42s
ztunnel-tvtzn                      1/1     Running   0             40s
ztunnel-vtpjm                      1/1     Running   0             40s
ztunnel-hllxg                      1/1     Running   0             40s

Repeat for each cluster

Repeat Upgrade CRDs and istiod and Upgrade the CNI and ztunnel for each cluster in your setup. Note that no multicluster components, such as the east-west gateway or global service entries, require upgrades when you update the version of your other control and data plane components.

Upgrade ambient meshes with Helm

Considerations link

Single cluster link

Upgrade istioctl link

Upgrade CRDs and istiod link

Upgrade the CNI and ztunnel link

Multicluster link

Upgrade istioctl link

Upgrade CRDs and istiod link

Upgrade the CNI and ztunnel link

Repeat for each cluster link

Considerations

Single cluster

Upgrade istioctl

Upgrade CRDs and istiod

Upgrade the CNI and ztunnel

Multicluster

Upgrade istioctl

Upgrade CRDs and istiod

Upgrade the CNI and ztunnel

Repeat for each cluster