☰
Navigation :

Upgrade gateway proxies

Upgrade the minor version of your unmanaged gateway proxies in workload clusters with zero downtime. This upgrade process uses a canary upgrade model in which you run two versions of Istio simultaneously.

Istio recommends upgrading by only one minor version at a time. If you want to upgrade multiple minor versions, such as from 1.14 to 1.16, upgrade incrementally from one version to the next minor version, such as first from 1.14 to 1.15, and then from 1.15 to 1.16.

The following steps upgrade the Istio architecture outlined in the Deploy Istio in production guide. This installation profile uses Istio revisions to facilitate the upgrade process, and deploys a gateway load balancer service that is not managed by Istio so that you can run multiple versions of one Istio gateway. For more information about the example resource files that are provided in the following steps, see the GitHub repository for Gloo use cases.

Step 1: Gather gateway proxy details and prepare revisions

Before you begin the upgrade process, take inventory of the Istio details and prepare for the new Istio revision.

  1. If you have not already, deploy the Bookinfo sample application with your existing Istio installations to keep track of changes during the upgrade.

  2. Verify that the version you want to upgrade to is tested and supported by the Solo distribution of Istio. If not, you might be able to upgrade Gloo Gateway, and then upgrade Istio.

  3. Check the Istio release notes for the upgrade version to prepare for any breaking changes.

  4. Note any resources that point to a specific Istio revision.

    • Custom Envoy filters: As a recommended practice, Envoy filters often target a specific Istio revision. When you create the Istio revision for the upgrade process, you might need to create filters that target the new Istio revision.
    • Namespaces: Note the Istio revisions that your service namespaces currently target. 
      kubectl get namespace -L istio.io/rev

      Example output:

      NAME              STATUS   AGE   REV
kube-system       Active   54m
default           Active   54m   1-19
bookinfo          Active   14s   1-19

  5. Save the existing and new Istio versions as environment variables. Versions are formatted such as 1.20.2-solo and revisions are formatted such as 1-20.

    • For REPO, use a Solo repo key for the new image that you can get by logging in to the Support Center and reviewing the Istio images built by Solo.io support article.
    • For ISTIO_IMAGE, save the new version that you want to install, such as 1.20.2, and append the solo tag. You can optionally append other tags for the Solo distribution of Istio, as described in About the Solo distribution of Istio. If you downloaded a different version than the following, make sure to specify that version instead.
    • For REVISION, Take the Istio major and minor versions and replace the period with a hyphen, such as 1-20.
    • For OLD_REVISION, take the Istio version for your existing version and replace the periods with hyphens, such as 1-19.
    REPO=<repo-key>
ISTIO_IMAGE=<upgrade_version>
REVISION=<upgrade_revision>
OLD_REVISION=<existing_revision>

  6. Multicluster setups only: Save the workload cluster that you want to upgrade Istio for as the environment variable CLUSTER_NAME so that you can reuse the same variable when you repeat these steps for all your workload clusters.

    export CLUSTER_NAME=$REMOTE_CLUSTER1

  7. In a separate terminal, generate some traffic to the Bookinfo app to verify that access to the app is uninterrupted and zero downtime occurs during the upgrade.

    1. Install the HTTP load testing utility.
    2. Get the address of the Istio ingress gateway. 
      
   export INGRESS_GW_IP=$(kubectl get svc -n gloo-mesh-gateways istio-ingressgateway-${OLD_REVISION} -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   echo http://$INGRESS_GW_IP/productpage
   
      
   export INGRESS_GW_IP=$(kubectl get svc -n gloo-mesh-gateways istio-ingressgateway-${OLD_REVISION} -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   echo http://$INGRESS_GW_IP/productpage
    3. Navigate to http://$INGRESS_GW_IP/productpage in a web browser to verify that the productpage for Bookinfo is reachable. 
      open http://$INGRESS_GW_IP/productpage
    4. Run the 15 minute load test that sends 10rps to localhost:8080/productpage. 
      RUN_TIME_SECONDS=900

echo "GET http://$INGRESS_GW_IP/productpage" | vegeta attack -rate 10/1s -duration=${RUN_TIME_SECONDS}s | vegeta encode > stats.json

vegeta report stats.json

vegeta plot stats.json > plot.html

Step 2: Deploy the new Istio control plane

Deploy the Istio control plane for the new version that you want to upgrade to.

  1. Download the Istio version that you want to upgrade to. The latest version supported by Solo, 1.20.2, is provided as an example.

    1. Download the new version.

      curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.20.2 sh -

    2. Navigate to the directory for the new version.

      cd ./istio-1.20.2

    3. Add the istioctl client to your path.

      export PATH=$PWD/bin:$PATH

  2. Update the Helm repository for Istio.

    helm repo update

  3. Upgrade the Istio CRDs.

    kubectl apply -f manifests/charts/base/crds

  4. Prepare a Helm values file for the istiod control plane. For example, you might update the revision, hub, and tag values in the existing values file that you used to install your current control plane. You can also use this sample command to download an example file, istiod.yaml, and update the environment variables with the values that you previously set.

    curl -0L https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/gloo-gateway/istio-install/manual-helm/istiod.yaml > istiod.yaml
envsubst < istiod.yaml > istiod-values.yaml

  5. Deploy the control plane that runs the new version to your cluster.

    helm upgrade --install istiod-${REVISION} istio/istiod \
  --version ${ISTIO_IMAGE} \
  --namespace istio-system \
  --wait \
  -f istiod-values.yaml

  6. After the installation is complete, verify that the Istio control plane pods for the new revision are now running alongside the existing pods for the old revision.

    kubectl get pods -n istio-system

    Example output:

    NAME                           READY   STATUS    RESTARTS   AGE
istiod-1-19-668dd8cc4c-6d49g   1/1     Running   0          42m
istiod-1-19-668dd8cc4c-btx8d   1/1     Running   0          42m
istiod-1-20-76fbc7b85c-7hh7f   1/1     Running   0          42s
istiod-1-20-76fbc7b85c-m5mlc   1/1     Running   0          41s

Step 3: Update gateways and validate traffic

Now that all the components for the new Istio version are deployed, you can deploy new Istio ingress and east-west gateways, and upgrade your apps’ Istio sidecars.

  1. Prepare a Helm values file for an ingress gateway that runs the new revision. For example, you might update the revision, hub, and tag values in the existing values file that you used to install your current gateway. You can also use this sample command to download an example file, ingress-gateway.yaml, and update the environment variables with the values that you previously set.

    curl -0L https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/gloo-gateway/istio-install/manual-helm/ingress-gateway.yaml > ingress-gateway.yaml
envsubst < ingress-gateway.yaml > ingress-gateway-values.yaml

  2. Create an ingress gateway that runs the new revision.

    helm upgrade --install istio-ingressgateway-${REVISION} istio/gateway \
  --version ${ISTIO_IMAGE} \
  --namespace gloo-mesh-gateways \
  --wait \
  -f ingress-gateway-values.yaml

    New load balancers are created for the canary gateways. To instead change the revision in use by the existing gateway load balancers, you can set the istio.io/rev label on the gateway deployment, which triggers a rolling restart.

  3. Multicluster setups only: Prepare a Helm values file for an east-west gateway that runs the new revision. For example, you might update the revision, hub, and tag values in the existing values file that you used to install your current gateway. You can also use this sample command to download an example file, eastwest-gateway.yaml, and update the environment variables with the values that you previously set.

    curl -0L https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/gloo-gateway/istio-install/manual-helm/eastwest-gateway.yaml > eastwest-gateway.yaml
envsubst < eastwest-gateway.yaml > eastwest-gateway-values.yaml

  4. Create an east-west gateway that runs the new revision.

    helm upgrade --install istio-eastwestgateway-${REVISION} istio/gateway \
  --version ${ISTIO_IMAGE} \
  --namespace gloo-mesh-gateways \
  --wait \
  -f eastwest-gateway-values.yaml

  5. Verify that the ingress and east-west gateway pods for the new revision are running alongside the existing gateways, and the load balancer services are assigned addresses.

    kubectl get pods -n gloo-mesh-gateways
kubectl get svc -n gloo-mesh-gateways

kubectl get pods -n gloo-mesh-gateways
kubectl get svc -n gloo-mesh-gateways

  6. Verify that the productpage for Bookinfo is still reachable after the upgrade.

    open http://$INGRESS_GW_IP/productpage

  7. Once the 15 minute load test is complete in your other terminal, check the results of the traffic requests that were sent to Bookinfo during the upgrade. The following example output shows that 6000 200 response codes and no error codes were received.

    Requests      [total, rate, throughput]         6000, 10.00, 10.00
Duration      [total, attack, wait]             15m0s, 15m0s, 26.776ms
Latencies     [min, mean, 50, 90, 95, 99, max]  15.344ms, 29.06ms, 25.727ms, 33.811ms, 41.936ms, 85.286ms, 1.212s
Bytes In      [total, mean]                     29091004, 4848.50
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:6000  
Error Set:

  8. You can also check the graph of the results that was generated.

    open ./plot.html

In the following example graph, a spike in latency occured when the Bookinfo application sidecars were updating. You might remedy this latency by adjusting the application scaling properties.

Example graph of the load test results

Step 4: Clean up previous resources

After you validate your upgrade, clean up the Istio resources that run the previous version.

  1. List the Istio Helm chart releases in the gloo-mesh-gateways namespace. In the output, find the name of your old revisions, such as istio-ingressgateway-1-19, and for multicluster setups, istio-eastwestgateway-1-19.

    helm ls -n gloo-mesh-gateways

  2. Delete the Helm release for the old ingress gateway, and for multicluster setups, the old east-west gateway.

    helm delete istio-ingressgateway-1-19 -n gloo-mesh-gateways
helm delete istio-eastwestgateway-1-19 -n gloo-mesh-gateways

  3. Verify that only the new gateways are running.

    kubectl get pods -n gloo-mesh-gateways

  4. List the Istio Helm chart releases in the istio-system namespace. In the output, find the name of your old revision, such as istiod-1-19.

    helm ls -n istio-system

  5. Delete the Helm release for the old control plane.

    helm delete istiod-1-19 -n istio-system

  6. Verify that only the control plane pods for the new revision are running.

    kubectl get pods -n istio-system

Step 6 (multicluster only): Repeat for each cluster

After you complete these steps, repeat the process for each workload cluster where you installed Istio. Be sure to change the CLUSTER_NAME environment variable to point to each workload cluster, and use each workload cluster's Kubernetes context.