Canary upgrades with the Gateway API

Perform a zero-downtime canary upgrade when your cluster uses Kubernetes Gateway API resources to configure ingress traffic.

Canary control-plane upgrades require extra care when gateways are managed by the Kubernetes Gateway API. This guide explains the tag-flip downtime issue, the versions that contain the fix, and the correct procedure for performing a zero-downtime canary upgrade.

About the tag-flip downtime issue

A canary upgrade runs two istiod control planes simultaneously and completes by pointing the default revision tag at the new revision. When that tag is flipped, the previously owning istiod re-evaluates which gateways it controls and determines it no longer owns them. In versions that do not include the fix for upstream issue #59959, the old istiod immediately sends an empty xDS push to every gateway pod still connected to it. Because the new istiod has not yet connected, gateway pods have no routing configuration for approximately 15–30 seconds, blackholing all traffic.

This issue does not affect upgrades that use in-place replacement rather than revision tags.

Versions with the fix

The fix is available in:

Upstream Istio 1.29.5 and later
Upstream Istio 1.30.2 and later
Upstream Istio 1.31.x (all patch versions)

The fix must be present in the currently-running version of istiod to take effect. If you are upgrading from a version that does not include the fix, old istiod still runs the unpatched code and clears xDS on the tag flip. The fix only protects canary upgrades performed from a version that already ships it.

Canary upgrades with the Gateway API are not safe on 1.30.0 or 1.30.1. To upgrade without downtime, first upgrade to 1.30.2 or later using the in-place Helm upgrade guide, then return to this guide to perform the canary upgrade.

Perform a zero-downtime canary upgrade

Warning

Do not set PILOT_ENABLE_GATEWAY_API=false on the old istiod during the migration. Disabling the Gateway API on the old control plane causes HTTPRoutes in namespaces not yet migrated to the canary revision to lose their status, because neither istiod will reconcile them. Keep the Gateway API enabled on both control planes for the duration of the migration.

Export your upgrade target version.

export NEW_VERSION=<new_istio_patch_version>
export NEW_REVISION=<revision_name>

Install the new istiod as a canary revision alongside the running control plane. For more information about the required and recommended Helm values for istiod, see the ambient installation guides for single cluster or multicluster setups.
```
helm install istiod-${NEW_REVISION} oci://us-docker.pkg.dev/soloio-img/istio-helm/istiod \
  --namespace istio-system \
  --version ${NEW_VERSION}-solo \
  --set revision=${NEW_REVISION}
```

Verify that the canary istiod is running.

kubectl get pods -n istio-system -l app=istiod

Migrate namespaces to the canary revision one at a time. For each namespace, relabel it with the new revision. When a namespace is relabeled, the canary istiod takes ownership of all Gateway and HTTPRoute resources in that namespace.
```
kubectl label namespace <namespace> istio.io/rev=${NEW_REVISION} --overwrite
```

Verify that Gateways and HTTPRoutes in the namespace receive status updates from the new istiod before proceeding to the next namespace.

kubectl get httproutes -n <namespace> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.parents[*].conditions[*].type}{"\n"}{end}'

After all namespaces are migrated, flip the default revision tag to point to the new revision. With the fix present in the running istiod, the old control plane continues serving xDS to any still-connected gateway pods for a grace period while the new istiod takes over. Traffic remains uninterrupted.
```
istioctl tag set default --revision ${NEW_REVISION} --overwrite
```
Verify that all gateways have reconnected to the new istiod and are receiving correct xDS configuration.
```
istioctl proxy-status | grep -E "gateway|ingress"
```
Remove the old istiod once all gateways show the new revision.
```
helm uninstall istiod -n istio-system
```

In-place upgrades with the Gloo Operator