Upgrading Gloo Mesh
Gloo Mesh is pre-1.0 and under active development. There may be breaking API changes with every minor version bump. An “API break” is defined as any change in the structure or behavior of a Gloo Mesh CRD that requires user intervention at upgrade time to avoid putting Gloo Mesh into a persistent error state.
The following steps describe a general approach to navigating the Gloo Mesh upgrade process for versions of Gloo Mesh < 1.0.0. This process involves scaling down the Gloo Mesh control plane while various components and CRDs are updated, but should not introduce downtime in service-to-service communication or policy enforcement because Gloo Mesh does not operate in the request path and the underlying service meshes (e.g. Istio) are not impacted by this process. That said, while Gloo Mesh is in its initial development ahead of general availability (i.e. pre-1.0.0), there is no guarantee that these steps will provide a comprehensive, zero-downtime upgrade experience. To reiterate:
1. Following the steps outlined by the Getting Started Guide,
download the version of
meshctl corresponding to the version of Gloo Mesh to which you would like to upgrade.
2. Scale down both the
discovery deployments on the Gloo Mesh management cluster to zero replicas.
This will prevent each component from processing resources structured in a way not expected by the new versions of each
kubectl scale deployment -n gloo-mesh networking --replicas 0
kubectl scale deployment -n gloo-mesh discovery --replicas 0
3. Delete all resources in the
discovery.gloo.mesh.gloo.solo.io API group such as
These resources will be recreated when
discovery is scaled back up later in the upgrade process. Deleting these
resources ensures that the latest discovery implementation will recreate them with the structure it expects.
4. Update the Gloo Mesh management plane CRDs in the management cluster. Note that if you will be upgrading your
control plane with
meshctl install, you can skip this step. Meshctl will apply the latest CRDs at upgrade time. If you
will be upgrading using Helm, read on.
Because of the way Helm handles CRDs at upgrade time, this step is required to ensure the management plane CRDs have the validation schemas expected by the version of Gloo Mesh you are upgrading to. This will prevent errors in the management plane and ensure that configuration leveraging the latest APIs on each Gloo Mesh are not rejected.
Gloo Mesh CRDs can be updated by extracting the contents of the Helm chart, for example https://storage.googleapis.com/gloo-mesh/gloo-mesh/gloo-mesh-0.11.3
0.11.3 is replaced by the version of Gloo Mesh to which you would like to upgrade. Alternatively, CRDs can be found
on GitHub at https://github.com/solo-io/gloo-mesh/tree/main/install/helm/gloo-mesh/crds.
From there, you can either download the repository and apply the CRDs at the tag of your choosing, or apply the CRDs
directly from GitHub with:
# Add your desired version here UPGRADE_VERSION=v0.11.3 kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/discovery.mesh.gloo.solo.io_v1alpha2_crds.yaml kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/multicluster.solo.io_v1alpha1_crds.yaml kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/networking.enterprise.mesh.gloo.solo.io_v1alpha1_crds.yaml kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/networking.mesh.gloo.solo.io_v1alpha2_crds.yaml kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/rbac.enterprise.mesh.gloo.solo.io_v1alpha1_crds.yaml kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh/$UPGRADE_VERSION/install/helm/gloo-mesh/crds/settings.mesh.gloo.solo.io_v1alpha2_crds.yaml
5. Upgrade the Gloo Mesh management plane components using either Helm or
meshctl, following the steps described in
the Gloo Mesh installation guide. Note that you
helm upgrade rather than
helm install if using Helm, and that
meshctl install can be used for both
install and upgrade operations.
6. Ensure that all
networking resources such as
conform to the latest APIs. Breaking changes will be highlighted in the changelog.
7. Re-register all clusters registered to the Gloo Mesh management plane. This will ensure that all remote cluster agents and CRDs are updated to a version compatible with the version of Gloo Mesh you are upgrading to. Refer to the setup guide on registering a cluster and be sure to use the same cluster names and contexts that were used at the initial cluster registration time.
8. Scale the
discovery deployment to one replica, and wait for all
discovery resources such as
workloads to be written to the management cluster. This may take a few minutes, and will ensure that the
component has access to all the data it needs to continue processing user-provided network configuration. Discovery is
complete when the pod no longer outputs a steady stream of logs or when all expected resources can be found on the cluster.
kubectl scale deployment -n gloo-mesh discovery --replicas 1
9. Scale the
networking deployment to one replica. Errors may be propagated to various
networking resources such as
accesspolicies as it starts watches on remote clusters,
but it will reach a steady state after a few moments.
kubectl scale deployment -n gloo-mesh networking --replicas 1
meshctl check to verify that all resources are in a healthy state. If not, check the logs on the
networking pods as well as the
status fields on unhealthy resources to begin debugging. Refer to our
Troubleshooting Guide for more details.