For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.
Release notes
Review summaries of the main changes in the 2.14 release.
Caution
Introduction
The release notes include important installation changes and known issues. They also highlight ways that you can take advantage of new features or enhancements to improve your product usage.
For more information, see the following related resources:
- Upgrade guide: Steps to upgrade from the previous minor version to the current version.
- Version reference: Information about Solo’s version support.
- Changelog: A full list of changes, including the ability to compare previous patch and minor versions.
đĨ Breaking changes
Review details about the following breaking changes. To review when breaking changes were released, you can use the comparison feature of the changelog. The severity is intended as a guide to help you assess how much attention to pay to this area during the upgrade, but can vary depending on your environment.
đ¨ High
Review severe changes that can impact production and require manual intervention.
- No high-severity changes are currently reported.
đ Medium
Review changes that might have impact to production and require manual intervention, but possibly not until the next version is released.
ext-auth-service strips claimsToHeaders and headersFromMetadata headers when source value is absent
Previously, when an OAuth2 claimsToHeaders mapping or an API-key headersFromMetadata mapping had no value to inject (because the claim was absent or the metadata entry was missing), ext-auth-service left the header untouched on the request. A client-supplied value for that header could pass through to the upstream, creating an identity injection risk. Starting in version 2.14, ext-auth-service strips the header when the source value is absent, preventing client-supplied values from bypassing headers that ext-auth-service owns.
Additionally, an append-mode claimsToHeaders mapping permanently switched to overwrite mode after its first use, which could overwrite a client-supplied header value or strip the header when the claim was subsequently absent. The mapping now consistently uses append mode across all requests.
If your deployment relies on a client-supplied header value passing through when the corresponding OAuth claim or metadata entry is absent, set PRESERVE_UNSET_INJECTED_HEADERS=true on the ext-auth-service Deployment to restore the previous behavior. Set the variable via extAuthService.extraEnvs in your Helm values:
extAuthService:
extraEnvs:
PRESERVE_UNSET_INJECTED_HEADERS:
value: "true"This setting re-enables the fail-open behavior that the fix addresses. Use it only as a temporary measure while updating upstreams that depend on the previous behavior.
OTel collector default image includes only components that Gloo Mesh requires
The bundled OTel collector image now includes 32 components, covering only what the built-in Gloo Mesh pipelines require. Previously, the image included approximately 190 components. This change reduces CVE exposure of the bundled image and makes future dependency updates straightforward.
This change does not affect standard Gloo Mesh observability. The built-in metrics, logs, traces, and insights pipelines all use components that remain in the default image.
If you set telemetryCollectorCustomization.extraReceivers, telemetryCollectorCustomization.extraProcessors, or telemetryCollectorCustomization.extraExporters in your Helm values with a component that is not in the default set, the collector fails to start after the upgrade and enters a crashloop.
To check whether this change affects you, review your Helm values for any extraReceivers, extraProcessors, or extraExporters entries, and verify that each component is in the following supported set.
| Category | Supported components |
|---|---|
| Extensions | healthcheck, basicauth, bearertokenauth |
| Receivers | otlp, nop, prometheus, filelog, jaeger, zipkin, redis |
| Processors | batch, memorylimiter, resource, filter, transform, metricstransform, attributes, k8sattributes |
| Exporters | otlp, otlphttp, debug, nop, prometheus, clickhouse, file, redisstream, datadog, elasticsearch, prometheusremotewrite, kafka, splunkhec, awss3 |
If you use a component outside this set, switch to the full-component fallback image by overriding the image tag in your Helm values:
telemetryCollector:
image:
tag: "0.5.0"
# For multicluster setups, also update the gateway:
telemetryGateway:
image:
tag: "0.5.0"The fallback image (gcr.io/gloo-mesh/otel-collector:0.5.0) is kept current with security patches and restores near-full component parity with the collector image that shipped with Gloo Mesh versions before 2.14 (gcr.io/gloo-mesh/otel-collector:0.3.9). The following components from the previous image are not included in the limited component or full-component image, because upstream removed them between contrib v0.128.0 and v0.158.0: carbon, loki, opencensus, sapm, bigip, jmx, ecstaskobserver. Of these, opencensus affects the most users, because it was compiled as both a receiver and an exporter and appears in many custom Gloo Mesh telemetry pipelines. If your pipeline references opencensus, update the configuration to remove or replace it before upgrading. Upstream also removed the routing processor. You can use the routingconnector replacement via the fallback image.
Wildcard hosts in an ExternalService now honor clientsideTls settings
In versions before 2.14, an ExternalService that used a wildcard host, such as *.example.com, ignored its clientsideTls settings. The proxy originated one-way TLS and verified the upstream against its own default trust bundle, no matter which settings you configured. Starting in version 2.14, a wildcard host honors clientsideTls when the port routes through an egress gateway that you select in egressGatewayRoutes. Exact hosts are unchanged.
Review the following two changes before you upgrade.
mode: DISABLE now takes effect. A wildcard host previously originated TLS even when you set mode: DISABLE. If a port reaches an HTTPS-only upstream only because of that behavior, set clientsideTls.mode to SIMPLE before you upgrade. Otherwise, the proxy stops originating TLS, and requests to that upstream fail.
Certificate files on a wildcard port that has no egress gateway now report an error. On a wildcard host, caCertificates, clientCertificate, and privateKey apply only when the port routes through an egress gateway. On any other wildcard port, these fields are ignored, as in previous versions. The ExternalService now reports a warning for caCertificates, and an error for clientCertificate and privateKey. Your traffic does not change when you upgrade. To clear the error, route the port through an egress gateway in egressGatewayRoutes. If the upstream uses a public certificate authority (CA) and requires no client certificate, remove the fields instead.
For more information about these fields, see the ExternalService API reference.
âšī¸ Low
Review informational updates that you might want to implement but that are unlikely to materially impact production.
Container images no longer include a shell
The following changes apply to version 2.14.1 and later.
The Gloo Mesh (Gloo Platform APIs) component images, such as the management server, agent, and UI images, use Chainguard distroless base images instead of Alpine and Ubuntu. The Redis image does not change. User IDs, ports, arguments, and mount paths do not change, and you do not need to change any Helm values to upgrade.
The images do not include a shell or a package manager, so commands such as kubectl exec <pod> -- sh no longer work in Gloo Mesh (Gloo Platform APIs) pods. The envoy container of the Gloo UI has a minimal shell for its entrypoint script, but no other tools, such as ls or cat. To debug a pod, use kubectl debug to attach an ephemeral container instead, such as in the following example.
kubectl debug -it -n <namespace> <pod> --image=busybox --target=<container>The meshctl debug report command now collects pod details through a port-forward instead of running commands inside the pods. Use the meshctl version that matches your Gloo Mesh (Gloo Platform APIs) version. An older meshctl version creates a report without the translator snapshots, metrics, and rate limiter configuration.
đ§ New known issues
No new known issues are currently reported.
đ New features
Review the following new features that are introduced in version 2.14 and that you can enable in your environment.
AllowedClientHeadersOnSuccess for HTTP passthrough auth
The HTTP passthrough auth configuration now supports an AllowedClientHeadersOnSuccess field. On a successful (200) passthrough auth response, headers in this list that are present on the auth server’s response are forwarded to the downstream client. Headers absent from the auth response are not added, and headers not in the list are never forwarded. When the field is unset, behavior is unchanged.
Istio 1.31 support
You can now run Gloo Mesh (Gloo Platform APIs) with Istio 1.31. Istio 1.26 is no longer supported. For more information, see the version support matrix, and the Solo distribution of Istio changelog for 1.31.
Mutual TLS origination for wildcard ExternalService hosts
An ExternalService that uses a wildcard host, such as *.partner.example.com, can now present a client certificate to the upstream when the port routes through an egress gateway. Set clientsideTls.mode to MUTUAL to use it. Previously, only an exact host presented a client certificate, so an upstream that required mutual TLS was reachable only if you created one ExternalService for each hostname. A wildcard host routes through an Envoy dynamic forward proxy rather than a DestinationRule, so some TLS settings still do not apply. For the configuration and the limitations, see Originate mutual TLS to a wildcard host.
đ Feature changes
Review the following changes that might impact how you use certain features in your environment.
maxConnectionDuration added to ConnectionPolicy TCP settings
You can now set maxConnectionDuration in the tcp block of a ConnectionPolicy resource to cap how long a TCP connection to the destination host remains open. Istio closes the connection after the duration elapses, regardless of whether the connection is actively used. The value must be at least 1ms and uses Go duration format, such as 300s or 1h. If the value is not set, there is no maximum duration. Values below 1ms are accepted by the API but are silently ignored.
New warnings on wildcard ExternalService and VirtualGateway resources
Configuration on a wildcard host that Gloo Mesh (Gloo Platform APIs) cannot apply is now reported on the resource instead of being dropped in silence. Routing and traffic do not change. After you upgrade, you might see a warning for the following cases:
- An unsupported TLS setting on a wildcard host:
credentialName, theISTIO_MUTUALmode, or atargetPortother than 443. - A
VirtualGatewaylistener that exposes both a wildcard host and an exact host that the wildcard covers. - Generated configuration for a namespace that approaches the Kubernetes object size limit. Reaching it needs 1000 or more distinct
clientsideTlsconfigurations on wildcard hosts.
For more information, see Block egress traffic with an egress gateway.
Bundled component versions updated to address CVEs
Version 2.14 updates the following bundled components and default image tags to address CVEs.
| Component | Previous version | Version 2.14 |
|---|---|---|
Prometheus server (prometheus.server.image.tag) | v3.9.1 | v3.13.2 |
prometheus-community/prometheus Helm chart | 28.9.1 | 29.25.0 |
| Prometheus config reloader | v0.89.0 | v0.93.1 |
| SPIRE server and VM agent | 1.12.1 | 1.12.6 |
ext-auth-service (extAuthService.extAuth.image.tag) | 0.84.0 | 0.86.2 |
rate-limiter (rateLimiter.rateLimiter.image.tag) | 0.18.15 | 0.21.0 |
No changes are needed for default installations. For custom installations, review the following details before you upgrade:
- Prometheus: Version 3.13.2 is the current upstream long-term support (LTS) release, with support through July 31, 2027. The image is
gcr.io/gloo-mesh/prometheus. If you mirror images to a private registry, mirror the new tag before you upgrade. The upgrade does not add Helm hooks, and theprometheus-server-migrationjob stays removed. Before you upgrade, compare any overrides in theprometheussection of your Helm values file against the values file for chart version 29.25.0. - SPIRE: The SPIRE server image and the VM agent SPIRE version stay in lockstep. If you pin
glooSpireServer.image.tagin your Helm values file, update it to 1.12.6 and rebuild your VM agent images. - ext-auth-service and rate-limiter: Only the default image tags change. If you pin these tags in your Helm values file, update them.
For the other dependency updates in this release, such as the Go toolchain, the labstack/echo library, and the Envoy base image for the Gloo UI, see the changelog.
Version 2.14.1 updates the following bundled components and default image tags to address CVEs. If you pin any of these tags in your Helm values file, update them.
| Component | Version 2.14.0 | Version 2.14.1 |
|---|---|---|
ext-auth-service (extAuthService.extAuth.image.tag) | 0.86.2 | 0.87.1 |
rate-limiter (rateLimiter.rateLimiter.image.tag) | 0.21.0 | 0.22.2 |
Redis for the rate limiter (rateLimiter.redis.image.tag) | 7.4.10-alpine | 7.4.11-alpine3.21 |
Prometheus server (prometheus.server.image.tag) | v3.13.2 | v3.13.5 |
đ Bug fixes
gRPC passthrough auth retries use exponential backoff
In an AuthConfig that uses a gRPC passthrough auth service, the retryPolicy retried the auth check at a constant interval. The interval always equaled retryBackOff.baseInterval, no matter which attempt it was. The retryBackOff.maxInterval value applied only to connection establishment, so it had no effect on the spacing between retries. Retries now use exponential backoff, and maxInterval caps the interval between attempts, as described in the AuthConfig API reference.
ext-auth-service no longer drops response headers in multi-config chains
In AuthConfig chains with multiple auth services, response_headers_to_add and headers_to_remove from an earlier successful auth service were discarded when a later successful service returned its own OkHttpResponse. These fields are now merged across the chain (concatenated, earlier service first). Additionally, when a user ID header was configured, rebuilding the OkHttpResponse to append the header dropped other fields, such as OIDC Set-Cookie headers and credential-header stripping. Both cases are fixed.
JWT policy translator fixed for gateways with dotted label selectors
In Gloo Mesh Enterprise versions before 2.14, the JWT policy translator serialized workload-key identifiers using a "." delimiter. A gateway Service selector containing label keys with dots (for example, mesh.example.com/channel) produced an ambiguous workload key that failed to parse, causing all JWTPolicy resources targeting that gateway to go to FAILED status and fail open. The workload key serialization now handles dotted label keys correctly.
Wildcard ExternalService traffic through an egress gateway no longer resets
Requests to a wildcard-host ExternalService that routed through an egress gateway could fail with a filter_chain_not_found reset. The client sidecar presented the fully qualified domain name (FQDN) of the egress gateway as the SNI value, which matched the generated listener only when that FQDN was also listed in the exposedExternalServices field of the VirtualGateway listener. The SNI value is now derived from the listener host that exposes the ExternalService.
đī¸ Removed features
Removed support for Istio 1.26
Istio 1.26 is no longer supported with Gloo Mesh (Gloo Platform APIs) version 2.14. For more information, see the version support matrix.
đ§ Known issues
The Solo team fixes bugs, delivers new features, and makes changes on a regular basis as described in the changelog. Some issues, however, might impact many users for common use cases. These known issues are as follows:
- Cluster names: Do not use underscores (
_) in the names of your clusters or in thekubeconfigcontext for your clusters. - Istio:
- If you use Istio versions 1.27.7, 1.28.4, 1.29.0 or later, and you install the Gloo Mesh (Gloo Platform APIs) management plane (Gloo UI) into a namespace other than
gloo-mesh, you must allow that namespace by listing it in theDEBUG_ENDPOINT_AUTH_ALLOWED_NAMESPACESenvironment variable of your istiod installation. For more information, see the release notes. - Due to a lack of support for the Istio CNI and iptables for the Istio proxy, you cannot run Istio (and therefore Gloo Mesh (Gloo Platform APIs)) on AWS Fargate. For more information, see the Amazon EKS issue.
- If you use Istio versions 1.27.7, 1.28.4, 1.29.0 or later, and you install the Gloo Mesh (Gloo Platform APIs) management plane (Gloo UI) into a namespace other than
- OTel pipeline: FIPS-compliant builds are not currently supported for the OTel collector agent image.
- Route name and matcher changes: When performing a bulk update for the name or matchers of a route in a RouteTable resource, the translation of the Istio VirtualService and EnvoyFilter might take some time to complete leading to policies temporarily not being applied to your routes. For more information about this issue and mitigation strategies, see Bulk route name and matcher updates.