Set up prompt guards

About prompt guards link

Prompt guards are mechanisms that ensure that prompt-based interactions with a language model are secure, appropriate, and aligned with the intended use. These mechanisms help to filter, block, monitor, and control LLM inputs and outputs to filter offensive content, prevent misuse, and ensure ethical and responsible AI usage.

With Gloo AI Gateway, you can set up prompt guards to block unwanted requests to the LLM provider and mask sensitive data. In this tutorial, you learn how to block any request with a credit card string in the request body and mask credit card numbers that are returned by the LLM.

Before you begin link

Complete the Authenticate with API keys tutorial.

Block unwanted requests link

Use the RouteOption resource and the promptGuard field to deny requests to the LLM provider that include the credit card string in the request body.

1. Update the RouteOption resource and add a custom prompt guard. The following example parses requests sent to the LLM provider and blocks them when the credit card string is detected in the request body. Requests are automatically denied with a 403 HTTP response code and a custom response message is returned.

      kubectl apply -f - <<EOF
    apiVersion: gateway.solo.io/v1
    kind: RouteOption
    metadata:
      name: openai-opt
      namespace: gloo-system
    spec:
      targetRefs:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute
        name: openai
      options:
        ai:
          promptGuard:
            request:
              customResponseMessage: "Rejected due to inappropriate content"
              matches:
              - "credit card"
    EOF
     

2. Send a request to the AI API that includes the string credit card in the request body. Verify that the request is denied with a 403 HTTP response code and the custom response message is returned.

     curl -v "$INGRESS_GW_ADDRESS:8080/openai" -H content-type:application/json -d '{
     "model": "gpt-3.5-turbo",
     "messages": [
       {
         "role": "user",
         "content": "Can you give me some examples of Master Card credit card numbers?"
       }
     ]
   }' | jq
     

   Example output:

     * Mark bundle as not supporting multiuse
   < HTTP/1.1 403 Forbidden
   < content-type: text/plain
   < date: Tue, 18 Jun 2024 04:48:18 GMT
   < server: envoy
   < transfer-encoding: chunked
   * Connection #0 to host 172.18.0.2 left intact
   Rejected due to inappropriate content
     

3. Send another request. This time, remove the word credit from the user prompt. Verify that the request now succeeds.

   info

   OpenAI is configured to not return any sensitive information, such as credit card or Social Security Numbers, even if they are fake. Because of that, the request does not return a list of credit card numbers.

     curl -v "$INGRESS_GW_ADDRESS:8080/openai" -H content-type:application/json -d '{
     "model": "gpt-3.5-turbo",
     "messages": [
       {
         "role": "user",
         "content": "Can you give me some examples of Master Card card numbers?"
       }
     ]
   }' | jq
     

   Example output:

     {
     "id": "chatcmpl-9bL6LqRHFi551X5kI7PPZgF0KIC2K",
     "object": "chat.completion",
     "created": 1718686277,
     "model": "gpt-3.5-turbo-0125",
     "choices": [
       {
         "index": 0,
         "message": {
           "role": "assistant",
           "content": "I apologize, but I am unable to provide examples of Master Card card numbers as it goes against the guidelines for handling sensitive information and could potentially lead to fraudulent activity. It is important to keep credit card information private and secure. If you need a card number for testing purposes, please use a dummy or test card number provided by card issuers for such purposes."
         },
         "logprobs": null,
         "finish_reason": "stop"
       }
     ],
     "usage": {
       "prompt_tokens": 19,
       "completion_tokens": 72,
       "total_tokens": 91
     },
     "system_fingerprint": null
   }
     

Mask sensitive data link

In the next step, you instruct the Gloo AI Gateway to mask credit card numbers that are returned by the LLM.

1. Add a response matcher to the RouteOption resource.

     kubectl apply -f - <<EOF
   apiVersion: gateway.solo.io/v1
   kind: RouteOption
   metadata:
     name: openai-opt
     namespace: gloo-system
   spec:
     targetRefs:
     - group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: openai
     options:
       ai:
        promptGuard:
           request:
             customResponseMessage: "Rejected due to inappropriate content"
             matches:
             - "credit card"
           response:
             matches:
             # Mastercard
             - '(?:^|\D)(5[1-5][0-9]{2}(?:\ |\-|)[0-9]{4}(?:\ |\-|)[0-9]{4}(?:\ |\-|)[0-9]{4})(?:\D|$)'
   EOF
     

2. Send another request to the AI API and include a fake VISA credit card number. Verify that the VISA number is detected and masked in your response.

     curl -v "$INGRESS_GW_ADDRESS:8080/openai" -H content-type:application/json -d '{
     "model": "gpt-3.5-turbo",
     "messages": [
       {
         "role": "user",
         "content": "What type of number is 5105105105105100?"
       }
     ]
   }' | jq
     

   Example output:

     {
     "id": "chatcmpl-9bLG8rKdpV9SYBC6GM8owBrlY5Xq6",
     "object": "chat.completion",
     "created": 1718686884,
     "model": "gpt-3.5-turbo-0125",
     "choices": [
       {
         "index": 0,
         "message": {
           "role": "assistant",
           "content": "XXXXXXXXXXX5100 is a 15-digit positive integer number."
         },
         "logprobs": null,
         "finish_reason": "stop"
       }
     ],
     "usage": {
       "prompt_tokens": 19,
       "completion_tokens": 14,
       "total_tokens": 33
     },
     "system_fingerprint": null
   }
     

Next link

Add further protection to your LLM provider by setting up rate limiting based on claims in a JWT token.