Several Gloo policies are implemented through Envoy filters in the gateway proxy. If the Envoy filter has an error, your traffic can be affected.

What’s happening

Some policies that depend on Envoy filters no longer take effect. For example, a request that previously had an external auth policy might stop requiring authentication. Even if you did not modify any policies or route tables, you might notice this behavior.

When you check the Gloo agent logs, you notice an error similar to the following:

  "msg":"failed upserting resource"
...
"err":"etcdserver: request is too large"

You might notice this behavior with one or more of following policies, which depend on Envoy filters:

  • CORS
  • CSRF
  • DLP
  • External auth
  • Fault injection
  • JWT
  • Rate limiting
  • Transformation
  • WAF

Why it’s happening

If you create many routes (> 500) with many large policies attached to the routes, the Kubernetes object size limitation for etcd can be reached.

How to fix it

  1. Upgrade your Gloo version to 2.3.16, 2.4.1, or 2.5.0 (when available) and later.
  2. As part of the upgrade or in a subsequent upgrade just for the management server, edit the Gloo management server deployment to enable the EXPERIMENTAL_SEGMENT_ENVOY_FILTERS_BY_MATCHER feature flag.
  
glooMgmtServer:
  env:
  - name: EXPERIMENTAL_SEGMENT_ENVOY_FILTERS_BY_MATCHER
    value: "true"
  - name: POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: POD_UID
    valueFrom:
      fieldRef:
        fieldPath: metadata.uid
  - name: K8S_MEM_LIMIT
    valueFrom:
      resourceFieldRef:
        divisor: "1"
        resource: limits.memory
  - name: LICENSE_KEY
    valueFrom:
      secretKeyRef:
        key: key
        name: gloo-mesh-enterprise-license
        optional: true
  - name: REDIS_USERNAME
    valueFrom:
      secretKeyRef:
        key: username
        name: redis-auth-secrets
        optional: true
  - name: REDIS_PASSWORD
    valueFrom:
      secretKeyRef:
        key: password
        name: redis-auth-secrets
        optional: true
 ```