Rate limit requests

About rate limiting link

Rate limiting on LLM provider token usage is primarily related to cost management, security and service stability. LLM providers charge based on the number of input (user prompts and system prompts) and output (responses from the model) tokens, which can make uncontrolled usage very expensive. With Gloo AI Gateway, you can configure rate limiting based on LLM usage so that organizations can enforce budget constraints across groups, teams, departments, and individuals, and ensure that their usage remains within predictable bounds. That way, you can avoid unexpected costs and prevent malicious attacks to your LLM provider.

In the following tutorial, you extract claims from the JWT tokens for Alice and Bob that you created in the Control access tutorial. Then, you enforce rate limits based on the values of their JWT token claims.

Before you begin link

Complete the Control access tutorial.

Set up rate limiting link

1. Create a RateLimitConfig with your rate limit rules. The following example sets a user limit of 70 tokens per hour. The user ID is extracted from the sub claim in the JWT token.

     kubectl apply -f- <<EOF
   apiVersion: ratelimit.solo.io/v1alpha1
   kind: RateLimitConfig
   metadata:
     name: per-user-counter
     namespace: gloo-system
   spec:
     raw:
       descriptors:
       - key: user-id
         rateLimit:
           requestsPerUnit: 70
           unit: HOUR
       rateLimits:
       - actions:
         - metadata:
             descriptorKey: user-id
             source: DYNAMIC
             default: unknown
             metadataKey:
               key: "envoy.filters.http.jwt_authn"
               path:
               - key: principal
               - key: sub
   EOF
     

2. Add the RateLimitConfig to your AIOption resource by using the spec.rateLimiting section.

     kubectl apply -f- <<EOF
   apiVersion: gateway.solo.io/v1
   kind: RouteOption
   metadata:
     name: openai-opt
     namespace: gloo-system
   spec:
     targetRefs:
     - group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: openai
     options:
       rateLimitConfigs:
         refs:
         - name: per-user-counter
           namespace: gloo-system
     

3. Send a request to the OpenAI endpoint and include the JWT token for Alice. Verify that the request succeeds.

     curl -v "$INGRESS_GW_ADDRESS:8080/openai" --header "Authorization: Bearer $ALICE_TOKEN" -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
      },
      {
        "role": "user",
        "content": "Compose a poem that explains the concept of recursion in programming."
      }
    ]
   }'
     

   Example output:

     {
     "id": "chatcmpl-9bLT1ofadlXEMpo53LcGjHsv3S5Ry",
     "object": "chat.completion",
     "created": 1718687683,
     "model": "gpt-3.5-turbo-0125",
     "choices": [
       {
         "index": 0,
         "message": {
           "role": "assistant",
           "content": "In the realm of code, a concept so divine,\nRecursion weaves patterns, like nature's design.\nA function that calls itself, with purpose and grace,\nIt solves problems complex, with elegance and pace.\n\nLike a mirror reflecting its own reflection,\nRecursion repeats with boundless affection.\nEach iteration holds a story untold,\nUnraveling mysteries, a journey unfold.\n\nInfinite loops, a dangerous abyss,\nRecursion beckons with a siren's sweet kiss.\nBase case in"
         },
         "logprobs": null,
         "finish_reason": "length"
       }
     ],
     "usage": {
       "prompt_tokens": 39,
       "completion_tokens": 100,
       "total_tokens": 139
     },
     "system_fingerprint": null
   }
     

4. Repeat the request. Verify that the request is now rate limited and that you get back a 429 HTTP response code, because only one 70 tokens per hour are allowed for a particular user.

     curl -v "$INGRESS_GW_ADDRESS:8080/openai" --header "Authorization: Bearer $ALICE_TOKEN" -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
      },
      {
        "role": "user",
        "content": "Compose a poem that explains the concept of recursion in programming."
      }
    ]
   }'
     

   Example output:

     * Mark bundle as not supporting multiuse
   < HTTP/1.1 429 Too Many Requests
   < x-envoy-ratelimited: true
   < date: Tue, 18 Jun 2024 05:15:13 GMT
   < server: envoy
   < content-length: 0