Release notes

Review summaries of the main changes in the 2.10 release.

Make sure that you review the breaking changes 🔥 that were introduced in this release and the impact that they have on your current environment.

Introduction

The release notes include important installation changes and known issues. They also highlight ways that you can take advantage of new features or enhancements to improve your product usage.

For more information, see the following related resources:

Upgrade guide: Steps to upgrade from the previous minor version to the current version.
Version reference: Information about Solo’s version support.

Changelog: A full list of changes, including the ability to compare previous patch and minor versions.

🔥 Breaking changes

Review details about the following breaking changes. To review when breaking changes were released, you can use the comparison feature of the changelog. The severity is intended as a guide to help you assess how much attention to pay to this area during the upgrade, but can vary depending on your environment.

🚨 High

Review severe changes that can impact production and require manual intervention.

Imported VirtualDestination client-side policies

The ImportedVirtualDestinationPolicyLegacyMode feature gate is added to let you temporarily keep client-side policy behavior when importing VirtualDestinations that do not have a backing service in the local cluster.

Previously, client-side policies were not properly applied to VirtualDestinations that were imported from one workspace to another and did not have a backing service in the local cluster.

This bug is fixed. Now by default, the importing behavior matches the expected behavior as described in the policy import docs.

The fix can impact the DestinationRules that are translated from the client-side policies as follows.

Many environments get additional DestinationRules to enforce the client-side policies that are now imported to the workspace.
Some environments might have modified or fewer translated DestinationRules from the client-side policies, such as if imported client-side policies result in fewer policies being applied from the importing workspace.

Legacy mode is deprecated, disabled by default in version 2.10, and planned to be removed in version 2.11.

Extauth denied auth response merging

The Gloo Mesh (Gloo Platform APIs) external auth server now merges denied authentication responses when a || boolean expression is used. This change allows you more control over returned responses. Previously, a 403 HTTP response code was returned in all cases. Use the following examples to learn more about the new behavior:

401 || 401 results in a 401 HTTP response code. Previously, this case returned a 403 HTTP response code.
401 || 403 results in a 403 response code.

You can disable the new behavior by enabling the following feature flag in your Helm values file:


extAuthService: 
  extraEnvs: 
    DONT_MERGE_DENIED_AUTH_RESPONSES: true

🔔 Medium

Review changes that might have impact to production and require manual intervention, but possibly not until the next version is released.

No medium-severity changes are currently reported.

ℹ️ Low

Review informational updates that you might want to implement but that are unlikely to materially impact production.

`otelmetricsprocessor` removed

The otelmetricsprocessor that was previously used to transform Cilium metrics is removed. In previous releases, the processor was removed from all Gloo Mesh (Gloo Platform APIs) components and not used anymore. If you have processes or automation that depend on the metrics labels that the processor created, make sure to update these accordingly.

Auto-generated extauth signing key

In version 2.10.7 and later, the extauth service is upgraded from version 0.74.1 to 0.80.1 to address HIGH and CRITICAL CVEs. The new version of extauth requires a valid signing key at all times and no longer accepts an empty signing key (""). When you upgrade to version 2.10.7, if no signing key is configured (no SIGNING_KEY environment variable is set on the Gloo UI), Gloo Mesh (Gloo Platform APIs) automatically creates a Kubernetes Secret that contains a randomly generated signing key. If you run multiple UI replicas, all replicas use this shared secret. If you already set a SIGNING_KEY environment variable, no action is required.

🚧 New known issues

Review new known issues and how to mitigate them.

Gloo UI discovery when moving from single to multicluster setups

In a single cluster setup, you might have installed the Gloo UI to collect telemetry data in your cluster. In this case, the Gloo UI component performs all service and mesh discovery. However, if you later decide to add this standalone cluster to a multicluster setup, you must create a Gloo agent in the cluster, which also performs service and mesh discovery. To prevent conflicts with discovery that the Gloo agent must now perform instead of the Gloo UI, you must first edit your existing Helm release for the Gloo UI to set glooUi.discovery.enabled to false before deploying the Gloo agent.

EnvoyFilter updates for gateway selectors

By default, Gloo Mesh (Gloo Platform APIs) uses the selector labels on the VirtualGateway and the workloadSelector labels on the RouteTable to generate the name of an EnvoyFilter. This way, Gloo Mesh (Gloo Platform APIs) can easily determine the gateway or sidecar proxy that must be patched.

If you update the label selectors on the VirtualGateway or RouteTable, the existing EnvoyFilters are deleted and recreated. In large environments or for VirtualGateways that have many routes attached, the recreation of the EnvoyFilters might take some time to complete and can lead to inconsistent configuration.

To avoid downtime, follow these steps when updating the label:

To avoid downtime for your app, it is recommended to perform a canary upgrade of your gateway deployment instead of updating the labels on the VirtualGateway directly.

Review the label selectors on your current gateway. For example, you might have a VirtualGateway that uses the following label selectors to select the ingress gateway:
```
app: istio-ingressgateway
istio: ingressgateway
release: prod
```
Deploy your new ingress gateway. You can use the steps in set up an ingress gateway to install the gateway. Make sure that the new gateway uses other labels than what your current gateway uses. For example, you might use the following labels:
```
app: istio-ingressgateway
istio: ingressgateway
release: canary
```

Create a temporary ClusterIP service that selects the new gateway deployment. For example, you can use the following service YAML configuration.

kubectl apply -f- <<EOF
apiVersion: v1
kind: Service
metadata:
  name: ingressgateway-temp
  namespace: istio-ingress
spec:
  type: ClusterIP
  ports:
  - name: http2
    port: 80
    targetPort: 8080
    protocol: TCP
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8443
  - name: tcp-status-port
    port: 15021
    protocol: TCP
    targetPort: 15021   
  selector:
    istio: ingressgateway
    app: istio-ingressgateway
    release: canary
EOF

Test your new gateway by using the ClusterIP address.
After you verified that the new gateway works as expected, remove the temporary CluserIP service.
```
kubectl delete service ingressgateway-temp -n istio-ingress
```
Change the labels on the ingress gateway deployment to match the labels of your current gateway. After the update, the load balancer service that exposes the current ingress gateway automatically starts sending traffic to the new ingress gateway.
```
kubectl patch deployment istio-ingressgateway \
  -n istio-ingress \
  --type='json' \
  -p='[{"op": "replace", "path": "/metadata/labels/release", "value":"prod"}]'
```
Remove the old ingress gateway.

To update the workload selectors on a RouteTable, use the following steps.

Scale down the Istio control plane istiod.

kubectl scale deployment istiod-$REVISION --replicas 0

Apply the label selector updates to your RouteTable.
Wait for the Gloo management server to recreate the new EnvoyFilters.
```
kubectl get envoyfilters -A
```
Scale up the Istio control plane again. Change the --replicas option to the number of replicas that you want.
```
kubectl scale deployment istio-$REVISION --replicas 1
```

🌟 New features

Review the following new features that are introduced in version 2.10 and that you can enable in your environment.

Istio 1.27 support

You can now run Gloo Mesh (Gloo Platform APIs) with Istio 1.27. Istio 1.22 is no longer supported. For more information, see the version support matrix, and the Solo distribution of Istio changelog for 1.27.

Virtual destination support in ExtAuthPolicy

Now, you can apply external auth policies to virtual destinations. For more information, see Apply external auth to virtual destinations.

`OPTIONAL_MUTUAL` setting on VirtualGateways

You can now configure your ingress gateway with the Istio OPTIONAL_MUTUAL TLS mode by using the VirtualGateway resource. Optional mutual is similar to the MUTUAL TLS mode, except that the client TLS certificate is optional. The client certificate is still requested during the TLS handshake. However, a TLS connection can still be established, even if the client does not present a certificate. If the client presents a certificate, it is validated by the server.

For more information, see the API docs.

`GLOO_CORS_TRANSLATION_MODE`

Gloo Mesh (Gloo Platform APIs) translates a RouteTable delegation tree with a root and all its child RouteTables into a single Istio VirtualService. In setups with large delegation trees, the size of the resulting VirtualService can quickly grow and reach the maximum size of 1.4MiB that the etcd data store can process, especially if route policies are also applied to these RouteTables. VirtualServices that exceed the maximum size are rejected by Kubernetes.

To reduce the size of the resulting VirtualService, you can now enable the GLOO_CORS_TRANSLATION_MODE environment variable on the Gloo management server in Gloo Mesh (Gloo Platform APIs). This environment variable changes the way CORS policies are translated. By default, CORS policies are added inline on the resulting VirtualService. With the GLOO_CORS_TRANSLATION_MODE setting, you can change this behavior and instead create a separate EnvoyFilter for the CORS policies.

To set the GLOO_CORS_TRANSLATION_MODE environment variable, perform an upgrade of your Gloo management server Helm installation. Add the following values to your Helm values file:


glooMgmtServer:
  extraEnvs:
    GLOO_CORS_TRANSLATION_MODE:
      value: "VS_AND_EF"

You can choose between the following values:

VS_ONLY: Add CORS policies inline on the VirtualService.
EF_ONLY: Add CORS policies as separate EnvoyFilters.
VS_AND_EF: Add CORS policies inline on the VirtualService and in separate EnvoyFilters.

If you want to change the way CORS policies are translated, first set the GLOO_CORS_TRANSLATION_MODE to VS_AND_EF so that the CORS policies are translated and added to VirtualServices and EnvoyFilters. This way, your apps remain protected with the CORS policy during the update. After the translation is finished, you can change the GLOO_CORS_TRANSLATION_MODE setting to VS_ONLY or EF_ONLY, depending on the translation mode that you chose.

Route to external TCP services

You can now route traffic to an external TCP service by using an Istio egress gateway. The mTLS connection from the pod is terminated at the egress gateway, and you can decide whether to forward TCP traffic or originate a new TLS connection to your external service.

For more information, see Route to external TCP services.

🔄 Feature changes

Review the following changes that might impact how you use certain features in your Gloo environment.

Deprecation of using Gloo Platform APIs to onboard external machines

Onboarding external workloads (VMs) by using Gloo Platform APIs is now deprecated. Consider migrating to an ambient mesh setup, in which you can onboard VMs to the mesh with a more streamlined method.

Upcoming end of support for the Istio lifecycle manager

Support for the Istio lifecycle manager, provided either by the istioInstallations section of the Helm chart or by the GatewayLifecycleManager and IstioLifecycleManager custom resources, ends in version 2.11, and the feature is removed in version 2.12.

Before version 2.11 is released, be sure to switch your Istio management to Helm, or to use the new way of installing managed Istio with the Gloo Operator. Check out the guides for installing ambient or sidecar meshes, or for migration steps, see Migrate to the Gloo Operator from the Istio lifecycle manager.

Updated Gloo UI design

The Gloo UI has an updated look and feel. All pages are available at-a-glance in the left-hand navigation. Additionally, the Gloo UI Graph now defaults to the new graph experience. To learn more about the new Gloo UI design, see Explore the UI.

🗑️ Removed features

Removed support for Istio 1.22

Istio 1.22 is no longer supported with Gloo Mesh (Gloo Platform APIs) version 2.10. For more information, see the version support matrix.

🚧 Known issues

The Solo team fixes bugs, delivers new features, and makes changes on a regular basis as described in the changelog. Some issues, however, might impact many users for common use cases. These known issues are as follows:

Cluster names: Do not use underscores (_) in the names of your clusters or in the kubeconfig context for your clusters.
Istio:
- If you use Istio versions 1.27.7, 1.28.4, 1.29.0 or later, and you install the Gloo Mesh (Gloo Platform APIs) management plane (Gloo UI) into a namespace other than gloo-mesh, you must allow that namespace by listing it in the DEBUG_ENDPOINT_AUTH_ALLOWED_NAMESPACES environment variable of your istiod installation. For more information, see the release notes.
- Patch versions 1.26.0 and 1.26.1 of the Solo distribution of Istio lack support for FIPS-tagged images. When upgrading or installing 1.26, be sure to use patch version 1.26.1-patch0 and later only.
- In the Solo distribution of Istio 1.25 and later, you can access enterprise-level features by passing your Solo license in the license.value or license.secretRef field of the Solo distribution of the istiod Helm chart. The Solo istiod Helm chart is strongly recommended due to the included safeguards, default settings, and upgrade handling to ensure a reliable and secure Istio deployment. Though it is not recommended, you can pass your license key in the open source istiod Helm chart by using the --set pilot.env.SOLO_LICENSE_KEY field.
- Istio patch versions 1.25.1 and 1.24.4 contain an upstream certificate rotation bug in which requests with more than one trusted root certificate cannot be validated. If you use Gloo Mesh (Gloo Platform APIs) to manage root certificate rotation and use Istio 1.25 or 1.24, be sure to use 1.25.2 or 1.24.5 and later only.
- Due to a lack of support for the Istio CNI and iptables for the Istio proxy, you cannot run Istio (and therefore Gloo Mesh (Gloo Platform APIs)) on AWS Fargate. For more information, see the Amazon EKS issue.
OTel pipeline: FIPS-compliant builds are not currently supported for the OTel collector agent image.
- Workspaces: If you run Istio version 1.21 or earlier and you reconfigure your Gloo workspaces, such as by moving from one workspace to multiple workspaces, routing to services that are exposed with a virtual destination might fail. You must re-apply the virtual destination to fix routing for these services. Note that this issue is fixed in Istio version 1.22 and later.
- Route name and matcher changes: When performing a bulk update for the name or matchers of a route in a RouteTable resource, the translation of the Istio VirtualService and EnvoyFilter might take some time to complete leading to policies temporarily not being applied to your routes. For more information about this issue and mitigation strategies, see Bulk route name and matcher updates.

Gloo Mesh (Gloo Platform APIs) changelog