Upgrading Gloo Mesh

Gloo Mesh is pre-1.0 and under active development. There may be breaking API changes with every minor version bump. An “API break” is defined as any change in the structure or behavior of a Gloo Mesh CRD that requires user intervention at upgrade time to avoid putting Gloo Mesh into a persistent error state.

The following steps describe a general approach to navigating the Gloo Mesh upgrade process for versions of Gloo Mesh < 1.0.0. This process involves scaling down the Gloo Mesh control plane while various components and CRDs are updated, but should not introduce downtime in service-to-service communication or policy enforcement because Gloo Mesh does not operate in the request path and the underlying service meshes (e.g. Istio) are not impacted by this process. That said, while Gloo Mesh is in its initial development ahead of general availability (i.e. pre-1.0.0), there is no guarantee that these steps will provide a comprehensive, zero-downtime upgrade experience. To reiterate:

1. Following the steps outlined by the Getting Started Guide, download the version of meshctl corresponding to the version of Gloo Mesh to which you would like to upgrade.

2. Scale down both the networking and discovery deployments on the Gloo Mesh management cluster to zero replicas. This will prevent each component from processing resources structured in a way not expected by the new versions of each component.

kubectl scale deployment -n gloo-mesh networking --replicas 0

kubectl scale deployment -n gloo-mesh discovery --replicas 0

3. Delete all resources in the discovery.gloo.mesh.gloo.solo.io API group such as meshes, destinations, and workloads. These resources will be recreated when discovery is scaled back up later in the upgrade process. Deleting these resources ensures that the latest discovery implementation will recreate them with the structure it expects.

4. Update the Gloo Mesh management plane CRDs in the management cluster. Note that if you will be upgrading your control plane with meshctl install, you can skip this step. Meshctl will apply the latest CRDs at upgrade time. If you will be upgrading using Helm, read on.

Because of the way Helm handles CRDs at upgrade time, this step is required to ensure the management plane CRDs have the validation schemas expected by the version of Gloo Mesh you are upgrading to. This will prevent errors in the management plane and ensure that configuration leveraging the latest APIs on each Gloo Mesh are not rejected.

Gloo Mesh CRDs can be updated by extracting the contents of the Helm chart, for example https://storage.googleapis.com/gloo-mesh/gloo-mesh/gloo-mesh-0.11.3 where 0.11.3 is replaced by the version of Gloo Mesh to which you would like to upgrade. Alternatively, CRDs can be found on GitHub at https://github.com/solo-io/gloo-mesh/tree/main/install/helm/gloo-mesh/crds. From there, you can either download the repository and apply the CRDs at the tag of your choosing, or apply the CRDs directly from GitHub with:

# Add your desired version here
UPGRADE_VERSION=v0.11.3

kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/discovery.mesh.gloo.solo.io_v1alpha2_crds.yaml
kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/multicluster.solo.io_v1alpha1_crds.yaml
kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/networking.enterprise.mesh.gloo.solo.io_v1alpha1_crds.yaml
kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/networking.mesh.gloo.solo.io_v1alpha2_crds.yaml
kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/rbac.enterprise.mesh.gloo.solo.io_v1alpha1_crds.yaml
kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/settings.mesh.gloo.solo.io_v1alpha2_crds.yaml

5. Upgrade the Gloo Mesh management plane components using either Helm or meshctl, following the steps described in the Gloo Mesh installation guide. Note that you should use helm upgrade rather than helm install if using Helm, and that meshctl install can be used for both install and upgrade operations.

6. Ensure that all networking resources such as trafficpolicies, virtualmeshes, failoverservices, or accesspolicies conform to the latest APIs. Breaking changes will be highlighted in the changelog.

7. Re-register all clusters registered to the Gloo Mesh management plane. This will ensure that all remote cluster agents and CRDs are updated to a version compatible with the version of Gloo Mesh you are upgrading to. Refer to the setup guide on registering a cluster and be sure to use the same cluster names and contexts that were used at the initial cluster registration time.

8. Scale the discovery deployment to one replica, and wait for all discovery resources such as meshes, destinations, and workloads to be written to the management cluster. This may take a few minutes, and will ensure that the networking component has access to all the data it needs to continue processing user-provided network configuration. Discovery is complete when the pod no longer outputs a steady stream of logs or when all expected resources can be found on the cluster.

kubectl scale deployment -n gloo-mesh discovery --replicas 1

9. Scale the networking deployment to one replica. Errors may be propagated to various networking resources such as trafficpolicies, virtualmeshes, failoverservices, and accesspolicies as it starts watches on remote clusters, but it will reach a steady state after a few moments.

kubectl scale deployment -n gloo-mesh networking --replicas 1

10. Run meshctl check to verify that all resources are in a healthy state. If not, check the logs on the discovery and networking pods as well as the status fields on unhealthy resources to begin debugging. Refer to our Troubleshooting Guide for more details.