Several Gloo policies are implemented through Envoy filters in the gateway proxy. If the Envoy filter has an error, your traffic can be affected.

What’s happening

Some policies that depend on Envoy filters no longer take effect. For example, a request that previously had an external auth policy might stop requiring authentication. Even if you did not modify any policies or route tables, you might notice this behavior.

When you check the Gloo agent logs, you notice an error similar to the following:

  "msg":"failed upserting resource"
"err":"etcdserver: request is too large"

You might notice this behavior with one or more of following policies, which depend on Envoy filters:

  • CORS
  • CSRF
  • DLP
  • External auth
  • Fault injection
  • JWT
  • Rate limiting
  • Transformation
  • WAF

Why it’s happening

Some Gloo policies, such as JWT or other external auth policies are translated into Envoy filters during the Gloo translation process. These Envoy filters are created per proxy and are then applied to the ingress gateway or sidecar proxy to enforce the policies. In environments where you apply policies to a lot of apps and routes, the size of the Envoy filter can become very large and exceed the maximum file size limit in etcd. When the maximum file size limit is reached, new configuration is rejected in etcd and Istio, which leads to policies not being applied and enforced properly.

How to fix it

To prevent this issue, the experimental environment variable EXPERIMENTAL_SEGMENT_ENVOY_FILTERS_BY_MATCHER was introduced in Gloo Mesh Enterprise version 2.3 and 2.4. If enabled on the Gloo management server, the server automatically breaks up large Envoy filters and creates an Envoy filter per matcher. If the environment variable is not set, Envoy filters are created per proxy.

Starting in version 2.5.0, the experimental environment variable is deprecated and its functionality is promoted to standard behavior. The Gloo management server now automatically creates Envoy filters for each matcher. If you did not previously enable the EXPERIMENTAL_SEGMENT_ENVOY_FILTERS_BY_MATCHER environment variable to create Envoy filters per matcher, your Envoy filters must be re-created as part of the upgrade to version 2.5.0.

If you want to enable this feature, use the upgrade steps in version 2.5 as a general guidance for how to safely scale down the Gloo management server, Gloo agent, and istiod, and re-create the Envoy filters in your environment.