Envoy filter policies too large
Several Gloo policies are implemented through Envoy filters in the gateway proxy. If the Envoy filter has an error, your traffic can be affected.
Some policies that depend on Envoy filters no longer take effect. For example, a request that previously had an external auth policy might stop requiring authentication. Even if you did not modify any policies or route tables, you might notice this behavior.
When you check the Gloo agent logs, you notice an error similar to the following:
"msg":"failed upserting resource" ... "err":"etcdserver: request is too large"
You might notice this behavior with one or more of following policies, which depend on Envoy filters:
- External auth
- Fault injection
- Rate limiting
Why it's happening
If you create many routes (> 500) with many large policies attached to the routes, the Kubernetes object size limitation for etcd can be reached.
How to fix it
Upgrade your Gloo version to 2.3.16, 2.4.1, or 2.5.0 (when available) and later.
As part of the upgrade or in a subsequent upgrade just for the management server, edit the Gloo management server deployment to enable the
EXPERIMENTAL_SEGMENT_ENVOY_FILTERS_BY_MATCHERfeature flag.Keep in mind that you have to include all of the environment variables to overwrite one. Helm does not merge values if you only provide one.
glooMgmtServer: env: - name: EXPERIMENTAL_SEGMENT_ENVOY_FILTERS_BY_MATCHER value: "true" - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_UID valueFrom: fieldRef: fieldPath: metadata.uid - name: K8S_MEM_LIMIT valueFrom: resourceFieldRef: divisor: "1" resource: limits.memory - name: LICENSE_KEY valueFrom: secretKeyRef: key: key name: gloo-mesh-enterprise-license optional: true - name: REDIS_USERNAME valueFrom: secretKeyRef: key: username name: redis-auth-secrets optional: true - name: REDIS_PASSWORD valueFrom: secretKeyRef: key: password name: redis-auth-secrets optional: true