Release notes
Review summaries of the main changes in the Gloo 2.5 release.
Introduction
The release notes include important installation changes and known issues. They also highlight ways that you can take advantage of new features or enhancements to improve your product usage.
For more information, see the following related resources:
- Upgrade guide: Steps to upgrade from the previous minor version to the current version.
- Version reference: Information about Solo’s version support.
Breaking changes
Review details about the following breaking changes.
Upstream Prometheus upgrade
Gloo Network includes a built-in Prometheus server to help monitor the health of your Gloo components. This release of Gloo upgrades the Prometheus community Helm chart from version 19.7.2 to 25.11.0. As part of this upgrade, upstream Prometheus changed the selector labels for the deployment, which requires recreating the deployment. To help with this process, the Gloo Helm chart includes a pre-upgrade hook that automatically recreates the Prometheus deployment during a Helm upgrade. This breaking change impacts upgrades from previous versions to version 2.4.10, 2.5.1, or 2.6.0 and later.
If you do not want the redeployment to happen automatically, you can disable this process by setting the prometheus.skipAutoMigration
Helm value to true
. For example, you might use Argo CD, which converts Helm pre-upgrade hooks to Argo PreSync
hooks and causes issues. To ensure that the Prometheus server is deployed with the right version, follow these steps:
- Confirm that you have an existing deployment of Prometheus at the old Helm chart version of
chart: prometheus-19.7.2
.kubectl get deploy -n gloo-mesh prometheus-server -o yaml | grep chart
- Delete the Prometheus deployment. Note that while Prometheus is deleted, you cannot observe Gloo performance metrics.
kubectl delete deploy -n gloo-mesh prometheus-server
- In your Helm values file, set the
prometheus.skipAutoMigration
field totrue
. - Continue with the Helm upgrade of Gloo Network. The upgrade recreates the Prometheus server deployment at the new version.
Prometheus annotations removed
In Gloo version 2.5.0, the prometheus.io/port: "<port_number>"
annotation was removed from the Gloo management server and agent. However, the prometheus.io/scrape: true
annotation is still present. If you have another Prometheus instance that runs in your cluster, and it is not set up with custom scraping jobs for the Gloo management server and agent, the instance automatically scrapes all ports on the management server and agent pods. This can lead to error messages in the management server and agent logs. Note that this issue is resolved in version 2.5.2. To resolve this issue in Gloo version 2.5.0 or 2.5.1, see Run another Prometheus instance alongside the built-in one.
Installation changes
In addition to comparing differences across versions in the changelog, review the following installation changes from the previous minor version to version 2.5.
Gloo agent health check port
Because you can now run the Gloo agent as a sidecar container in the management server pod, the default Gloo agent health check port is changed from 8090 to 8091.
Gloo UI
To use the Gloo UI and visualize the network traffic in your environment with the Gloo UI graph, you must set the telemetryCollector.enabled
Helm setting to true
in each cluster in your environment, including the management cluster.
Portal logs pipeline
The Gloo telemetry pipeline telemetryCollectorCustomization.pipelines.logs/istio_access_logs
is renamed to telemetryCollectorCustomization.pipelines.logs/portal
. For more information, see Monitor Portal analytics in the Gloo Mesh Gateway docs.
Bug fixes
Multiple Istio revisions in the same cluster
If you run multiple revisions of Istio in your cluster and use discoverySelectors
in each revision to discover the resources in specific namespaces, enable the glooMgmtServer.extraEnvs.IGNORE_REVISIONS_FOR_VIRTUAL_DESTINATION_TRANSLATION
environment variable on the Gloo management server. This setting allows virtual destinations to be translated correctly if the east-west gateway and the backing services belong to different namespaces.
This feature is available in version 2.5.5 and later.
To enable this feature, add the following values to your Helm values file.
glooMgmtServer:
extraEnvs:
- name: IGNORE_REVISIONS_FOR_VIRTUAL_DESTINATION_TRANSLATION
value: "true"
To check if you use discoverySelectors
in your Istio revision:
Istio lifecycle manager installations:
Get the details of your Istio lifecycle manager resources.
kubectl get istiolifecyclemanagers -A -o yaml
In your Istio lifecycle manager resource, check if you use
discoverySelectors
in yourspec.installations.istioOperatorSpec.meshConfig
... spec: installations: - clusters: - defaultRevision: true name: mycluster istioOperatorSpec: components: pilot: k8s: env: - name: PILOT_SKIP_VALIDATE_TRUST_DOMAIN value: "true" meshConfig: discoverySelectors: - matchLabels: istio-discovery: enabled
Manual installations:
Get the details of your Istio operator.
kubectl get istiooperator -A -o yaml
In your Istio operator configuration, check if you use
discoverySelectors
in yourmeshConfig
settings.... meshConfig: discoverySelectors: - matchLabels: istio-discovery: enabled
New features
Review the following new features that are introduced in version 2.5 and that you can enable in your environment.
Redis safe mode
In versions 2.5.3 and lower, a race condition was identified that can be triggered during simultaneous restarts of the management plane and Redis, including an upgrade to a newer Gloo Mesh Enterprise version. If hit, this failure mode can lead to partial translations on the Gloo management server which can result in Istio resources being temporarily deleted from the output snapshots that are sent to the Gloo agents. For more information about this failure scenario, see Redis and Gloo management server restart. To resolve this issue, a new safe mode feature was added that you can enable by setting glooMgmtServer.safeMode
Helm chart option to true.
If safe mode is enabled, translation of input snapshots halts until the input snapshots of all registered Gloo agents are present in the Redis cache. This feature improves management plane stability during disaster scenarios and upgrades. For more information, see Safe mode. The safe mode feature is disabled by default.
Redis safe start window
With safe mode, the Gloo management server halts translation until the input snapshots of all workload clusters are present in the Redis cache. However, if clusters have connectivity issues, translation might be halted for a long time, even for healthy clusters. You might want translation to resume after a certain period of time, even if some input snapshots are missing in the Redis cache. To do so, you must use the glooMgmtServer.safeStartWindow
field in your Gloo management server Helm values file to specify the time in seconds to halt translation. Note that this setting is ignored if glooMgmtServer.safeMode
is set to true. The default value is 180 seconds. You can disable the wait time by setting this field to 0
(zero). For more information, see Option 2: Safe start window.
I/O threads for Redis in 2.5.6
A new Helm value redis.deployment.ioThreads
was introduced to specify the number of I/O threads to use for the built-in Redis instance. Redis is mostly single threaded, however some operations, such as UNLINK or slow I/O accesses can be performed on side threads. Increasing the number of side threads can help improve and maximize the performance of Redis as these operations can run in parallel.
The default and minimum valid value for this setting is 1. If you plan to increase the number of I/O side threads, make sure that you also change the CPU requests and CPU limits for the Redis pod. Set the CPU requests and limits to the same number that you use for the I/O side threads plus 1. That way, you can ensure that each side thread has an available CPU core, and that an additional CPU core is left for the main Redis thread. For example, if you want to set I/O threads to 2, make sure to add 3 CPU cores to the resource requests and limits for the Redis pod. You can find further recommendations regarding I/O threads in this Redis configuration example.
If you set I/O threads, the Redis pod must be restarted during the upgrade so that the changes can be applied. During the restart, the input snapshots from all connected Gloo agents are removed from the Redis cache. If you also update settings in the Gloo management server that require the management server pod to restart, the management server’s local memory is cleared and all Gloo agents are disconnected. Although the Gloo agents attempt to reconnect to send their input snapshots and re-populate the Redis cache, some agents might take longer to connect or fail to connect at all. To ensure that the Gloo management server halts translation until the input snapshots of all workload cluster agents are present in Redis, it is recommended to enable safe mode on the management server alongside updating the I/O threads for the Redis pod. For more information, see Safe mode. Note that in version 2.6.0 and later, safe mode is enabled by default.
To update I/O side threads in Redis as part of your Gloo Mesh Enterprise upgrade:
Scale down the number of Gloo management server pods to 0.
kubectl scale deployment gloo-mesh-mgmt-server --replicas=0 -n gloo-mesh
Upgrade Gloo Mesh Enterprise and use the following settings in your Helm values file for the management server. Make sure to also increase the number of CPU cores to one core per thread, and add an additional CPU core for the main Redis thread. The following example also enables safe mode on the Gloo management server to ensure translation is done with the complete context of all workload clusters.
glooMgmtServer: safeMode: true redis: deployment: ioThreads: 2 resources: requests: cpu: 3 limits: cpu: 3
Known issues
The Solo team fixes bugs, delivers new features, and makes changes on a regular basis as described in the changelog. Some issues, however, might impact many users for common use cases. These known issues are as follows:
- Cluster names: Do not use underscores (
_
) in the names of your clusters or in thekubeconfig
context for your clusters. - OTel pipeline: FIPS-compliant builds are not currently supported for the OTel collector agent image.