For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.
Global rate limiting
Apply distributed rate limits across multiple agentgateway replicas using the enterprise rate limit service.
Apply distributed rate limits across multiple agentgateway replicas using the built-in enterprise rate limit service.
About
Global rate limiting coordinates rate limits across multiple agentgateway proxy replicas. Unlike local rate limiting, which runs independently on each proxy replica, global rate limiting provides:
- Shared counters across all proxy replicas
- Consistent enforcement regardless of which replica receives the request
- Descriptor-based limits that can extract multiple request attributes using CEL expressions
- Flexible targeting by user, API key, IP address, path, or any combination
Global rate limiting is essential when running multiple proxy replicas and you need to enforce a single quota across the entire fleet — for example, “100 requests per minute per user” should apply to the sum of requests across all replicas, not 100 per replica.
Solo Enterprise for agentgateway includes a built-in rate limit service, so you do not need to deploy your own Redis or rate limit service infrastructure. Global rate limiting requires two components:
RateLimitConfig: A custom resource (
ratelimit.solo.io/v1alpha1) that defines the rate limit rules, including descriptors, limits, and CEL-based actions.EnterpriseAgentgatewayPolicy with
entRateLimit: References one or more RateLimitConfig resources to apply the limits to specific routes or the entire gateway.
Note
To use a BYO rate limit service instead of the built-in enterprise rate limit service, see the agentgateway docs.
For rate limiter and Redis connections and ports, see Rate limiting and budgets deployment.
Request flow
Global rate limiting works as follows:
- A CEL expression in the RateLimitConfig extracts request attributes (such as client IP, user ID, or path)
- The gateway sends these descriptor key-value pairs to the enterprise rate limit service
- The rate limit service matches the descriptors against the RateLimitConfig rules and checks the counter
- If the limit is exceeded, the gateway returns a 429 response to the client; otherwise, the request proceeds to the backend
Common CEL expressions
Review the following common CEL expressions that you might find useful when creating your rate limit policies.
| Descriptor | CEL Expression | Description |
|---|---|---|
| Client IP | source.address | Source IP address of the client. |
| Request path | request.path | The request path (such as /api/v1/users). |
| Request method | request.method | HTTP method (GET, POST, etc.). |
| Header value | request.headers["header-name"] | Extract a specific header. Case-insensitive header name. |
| JWT claim | claims["sub"] | Extract a claim from a validated JWT (requires JWT auth policy). |
| Static value | "constant" | Use a constant string for categorization (such as service tier). |
For an overview of supported CEL expressions, see the agentgateway docs.
Bring your own Redis
By default, the rate limiter stores counters in an in-cluster Redis instance, the ext-cache-enterprise-agentgateway extension. This instance does not use persistent storage, so counters reset when its pod is rescheduled. For production, use an external Redis instance. For your options and the steps, see Use an external Redis.
Before you begin
Rate limit by client IP
Limit requests based on the client’s source IP address.
Create a RateLimitConfig that limits each client IP to 10 requests per minute.
kubectl apply -f- <<EOF apiVersion: ratelimit.solo.io/v1alpha1 kind: RateLimitConfig metadata: name: ip-rate-limit namespace: agentgateway-system spec: raw: descriptors: - key: remote_address rateLimit: requestsPerUnit: 10 unit: MINUTE rateLimits: - actions: - cel: expression: 'source.address' key: "remote_address" EOFApply the rate limit by using an EnterpriseAgentgatewayPolicy resource. The following example targets all routes through the gateway.
kubectl apply -f- <<EOF apiVersion: enterpriseagentgateway.solo.io/v1alpha1 kind: EnterpriseAgentgatewayPolicy metadata: name: ip-rate-limit namespace: agentgateway-system spec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: agentgateway-proxy traffic: entRateLimit: global: rateLimitConfigRefs: - name: ip-rate-limit EOFVerify that the policy is attached.
kubectl get EnterpriseAgentgatewayPolicy ip-rate-limit -n agentgateway-system -o yamlBoth
AcceptedandAttachedconditions must beTrue.Send 15 requests and verify that the rate limit is enforced after 10 requests.
for i in $(seq 1 15); do STATUS=$(curl -s -o /dev/null -w "%{http_code}" \ http://$INGRESS_GW_ADDRESS/openai \ -H content-type:application/json \ -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hi"}]}') echo "Request $i: HTTP $STATUS" donefor i in $(seq 1 15); do STATUS=$(curl -s -o /dev/null -w "%{http_code}" \ localhost:8080/openai \ -H content-type:application/json \ -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hi"}]}') echo "Request $i: HTTP $STATUS" doneThe first 10 requests succeed with 200, and subsequent requests return 429.
Example output:
Request 1: HTTP 200 Request 2: HTTP 200 ... Request 10: HTTP 200 Request 11: HTTP 429 Request 12: HTTP 429 ... Request 15: HTTP 429
Rate limit by user ID
Extract the user ID from a request header and rate limit per user.
Create a RateLimitConfig that limits each user to 50 requests per minute.
kubectl apply -f- <<EOF apiVersion: ratelimit.solo.io/v1alpha1 kind: RateLimitConfig metadata: name: user-rate-limit namespace: agentgateway-system spec: raw: descriptors: - key: x-user-id rateLimit: requestsPerUnit: 50 unit: MINUTE rateLimits: - actions: - cel: expression: 'request.headers["x-user-id"]' key: "x-user-id" EOFApply the rate limit by using a EnterpriseAgentgatewayPolicy resource. The following example targets the
openaiHTTPRoute.kubectl apply -f- <<EOF apiVersion: enterpriseagentgateway.solo.io/v1alpha1 kind: EnterpriseAgentgatewayPolicy metadata: name: user-rate-limit namespace: agentgateway-system spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: openai traffic: entRateLimit: global: rateLimitConfigRefs: - name: user-rate-limit EOFSend requests with a user ID header to trigger the rate limit.
for i in $(seq 1 60); do STATUS=$(curl -s -o /dev/null -w "%{http_code}" \ http://$INGRESS_GW_ADDRESS/openai \ -H content-type:application/json \ -H "x-user-id: alice" \ -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hi"}]}') echo "Request $i: HTTP $STATUS" donefor i in $(seq 1 60); do STATUS=$(curl -s -o /dev/null -w "%{http_code}" \ localhost:8080/openai \ -H content-type:application/json \ -H "x-user-id: alice" \ -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hi"}]}') echo "Request $i: HTTP $STATUS" doneThe first 50 requests succeed with 200, and subsequent requests return 429. Each unique user ID gets its own rate limit counter.
Example rate-limited response:
< HTTP/1.1 429 Too Many Requests < x-envoy-ratelimited: true < date: Tue, 18 Jun 2024 05:15:13 GMT < server: envoy < content-length: 0
Combine local and global rate limiting
Apply both local and global rate limits to the same traffic.
Create a RateLimitConfig for the global per-user limit.
kubectl apply -f- <<EOF apiVersion: ratelimit.solo.io/v1alpha1 kind: RateLimitConfig metadata: name: combined-global-limit namespace: agentgateway-system spec: raw: descriptors: - key: x-user-id rateLimit: requestsPerUnit: 50 unit: MINUTE rateLimits: - actions: - cel: expression: 'request.headers["x-user-id"]' key: "x-user-id" EOFApply two EnterpriseAgentgatewayPolicy resources that target the same HTTPRoute, one for the local limit and one for the global limit. A single EnterpriseAgentgatewayPolicy cannot set both
traffic.rateLimitandtraffic.entRateLimit.kubectl apply -f- <<EOF apiVersion: enterpriseagentgateway.solo.io/v1alpha1 kind: EnterpriseAgentgatewayPolicy metadata: name: combined-rate-limit-local namespace: agentgateway-system spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: openai traffic: rateLimit: local: - requests: 100 unit: Seconds --- apiVersion: enterpriseagentgateway.solo.io/v1alpha1 kind: EnterpriseAgentgatewayPolicy metadata: name: combined-rate-limit-global namespace: agentgateway-system spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: openai traffic: entRateLimit: global: rateLimitConfigRefs: - name: combined-global-limit EOFThis configuration enforces:
- Local: 100 requests per second per proxy replica (protects against traffic spikes)
- Global: 50 requests per minute per user across all replicas (enforces user quotas)
Both limits are evaluated. A request must pass both checks to succeed.
Cleanup
You can remove the resources that you created in this guide.kubectl delete RateLimitConfig ip-rate-limit -n agentgateway-system
kubectl delete RateLimitConfig user-rate-limit -n agentgateway-system
kubectl delete RateLimitConfig combined-global-limit -n agentgateway-system
kubectl delete EnterpriseAgentgatewayPolicy ip-rate-limit -n agentgateway-system
kubectl delete EnterpriseAgentgatewayPolicy user-rate-limit -n agentgateway-system
kubectl delete EnterpriseAgentgatewayPolicy combined-rate-limit-local -n agentgateway-system
kubectl delete EnterpriseAgentgatewayPolicy combined-rate-limit-global -n agentgateway-system