Ai

Package: ai.options.gloo.solo.io

Types:

• SingleAuthToken
• Passthrough
• UpstreamSpec
• CustomHost
• OpenAI
• AzureOpenAI
• Gemini
• VertexAI
• Publisher
• Mistral
• Anthropic
• MultiPool
• Backend
• Priority
• RouteSettings
• RouteType
• FieldDefault
• Postgres
• Embedding
• OpenAI
• AzureOpenAI
• SemanticCache
• Redis
• Weaviate
• DataStore
• Mode
• RAG
• DataStore
• AIPromptEnrichment
• Message
• AIPromptGuard
• Regex
• RegexMatch
• BuiltIn
• Action
• Webhook
• HeaderMatch
• MatchType
• Moderation
• OpenAI
• Request
• CustomResponse
• Response

Source File: github.com/solo-io/gloo/projects/gloo/api/v1/enterprise/options/ai/ai.proto

SingleAuthToken

The authorization token that the AI gateway uses to access the LLM provider API. This token is automatically sent in a request header, depending on the LLM provider.

"inline": string
"secretRef": .core.solo.io.ResourceRef
"passthrough": .ai.options.gloo.solo.io.SingleAuthToken.Passthrough

Passthrough

Configuration for passthrough of the existing token. Currently, specifying an empty object (passthrough: {}) indicates that passthrough will be used for auth.

UpstreamSpec

When you deploy the Gloo AI Gateway, you can use the spec.ai section of the Upstream resource to represent a backend for a logical Large Language Model (LLM) provider. This section configures the LLM provider that the AI Gateway routes requests to, and how the gateway should authenticate with the provider. Note that other Gloo AI Gateway LLM features, such as prompt guards and prompt enrichment, are configured at the route level in the spec.options.ai section of the RouteOptions resource.

To get started, see About Gloo AI Gateway. For more information about the Upstream resource, see the API reference.

"openai": .ai.options.gloo.solo.io.UpstreamSpec.OpenAI
"mistral": .ai.options.gloo.solo.io.UpstreamSpec.Mistral
"anthropic": .ai.options.gloo.solo.io.UpstreamSpec.Anthropic
"azureOpenai": .ai.options.gloo.solo.io.UpstreamSpec.AzureOpenAI
"multi": .ai.options.gloo.solo.io.UpstreamSpec.MultiPool
"gemini": .ai.options.gloo.solo.io.UpstreamSpec.Gemini
"vertexAi": .ai.options.gloo.solo.io.UpstreamSpec.VertexAI

CustomHost

Send requests to a custom host and port, such as to proxy the request, or to use a different backend that is API-compliant with the upstream version.

"host": string
"port": int

OpenAI

Settings for the OpenAI LLM provider.

"authToken": .ai.options.gloo.solo.io.SingleAuthToken
"customHost": .ai.options.gloo.solo.io.UpstreamSpec.CustomHost
"model": string

AzureOpenAI

Settings for the Azure OpenAI LLM provider. To find the values for the endpoint, deployment name, and API version, you can check the fields of an API request, such as https://{endpoint}/openai/deployments/{deployment_name}/chat/completions?api-version={api_version}.

"authToken": .ai.options.gloo.solo.io.SingleAuthToken
"endpoint": string
"deploymentName": string
"apiVersion": string

Gemini

Settings for the Gemini LLM provider. To find the values for the model and API version, you can check the fields of an API request, such as https://generativelanguage.googleapis.com/{version}/models/{model}:generateContent?key={api_key}.

"authToken": .ai.options.gloo.solo.io.SingleAuthToken
"model": string
"apiVersion": string

VertexAI

Settings for the Vertex AI LLM provider. To find the values for the project ID, project location, and publisher, you can check the fields of an API request, such as https://{LOCATION}-aiplatform.googleapis.com/{VERSION}/projects/{PROJECT_ID}/locations/{LOCATION}/publishers/{PROVIDER}/<model-path>.

"authToken": .ai.options.gloo.solo.io.SingleAuthToken
"model": string
"apiVersion": string
"projectId": string
"location": string
"modelPath": string
"publisher": .ai.options.gloo.solo.io.UpstreamSpec.VertexAI.Publisher

Publisher

The type of publisher model to use. Currently, only Google is supported.

Mistral

Settings for the Mistral AI LLM provider.

"authToken": .ai.options.gloo.solo.io.SingleAuthToken
"customHost": .ai.options.gloo.solo.io.UpstreamSpec.CustomHost
"model": string

Anthropic

Settings for the Anthropic LLM provider.

"authToken": .ai.options.gloo.solo.io.SingleAuthToken
"customHost": .ai.options.gloo.solo.io.UpstreamSpec.CustomHost
"version": string
"model": string

MultiPool

Configure backends for multiple hosts or models from the same provider in one Upstream resource. This method can be useful for creating one logical endpoint that is backed by multiple hosts or models.

In the priorities section, the order of pool entries defines the priority of the backend endpoints. The pool entries can either define a list of backends or a single backend. Note: Only two levels of nesting are permitted. Any nested entries after the second level are ignored.

multi:
  priorities:
  - pool:
    - azureOpenai:
        deploymentName: gpt-4o-mini
        apiVersion: 2024-02-15-preview
        endpoint: ai-gateway.openai.azure.com
        authToken:
          secretRef:
            name: azure-secret
            namespace: gloo-system
  - pool:
    - azureOpenai:
        deploymentName: gpt-4o-mini-2
        apiVersion: 2024-02-15-preview
        endpoint: ai-gateway-2.openai.azure.com
        authToken:
          secretRef:
            name: azure-secret-2
            namespace: gloo-system

"priorities": []ai.options.gloo.solo.io.UpstreamSpec.MultiPool.Priority

Backend

An entry represeting an LLM provider backend that the AI Gateway routes requests to.

"openai": .ai.options.gloo.solo.io.UpstreamSpec.OpenAI
"mistral": .ai.options.gloo.solo.io.UpstreamSpec.Mistral
"anthropic": .ai.options.gloo.solo.io.UpstreamSpec.Anthropic
"azureOpenai": .ai.options.gloo.solo.io.UpstreamSpec.AzureOpenAI
"gemini": .ai.options.gloo.solo.io.UpstreamSpec.Gemini
"vertexAi": .ai.options.gloo.solo.io.UpstreamSpec.VertexAI

Priority

The order of pool entries within this section defines the priority of the backend endpoints.

"pool": []ai.options.gloo.solo.io.UpstreamSpec.MultiPool.Backend

RouteSettings

When you deploy the Gloo AI Gateway, you can use the spec.options.ai section of the RouteOptions resource to configure the behavior of the LLM provider on the level of individual routes. These route settings, such as prompt enrichment, retrieval augmented generation (RAG), and semantic caching, are applicable only for routes that send requests to an LLM provider backend.

For more information about the RouteOptions resource, see the API reference.

"promptEnrichment": .ai.options.gloo.solo.io.AIPromptEnrichment
"promptGuard": .ai.options.gloo.solo.io.AIPromptGuard
"rag": .ai.options.gloo.solo.io.RAG
"semanticCache": .ai.options.gloo.solo.io.SemanticCache
"defaults": []ai.options.gloo.solo.io.FieldDefault
"routeType": .ai.options.gloo.solo.io.RouteSettings.RouteType

RouteType

The type of route to the LLM provider API.

FieldDefault

Provide defaults to merge with user input fields. Defaults do not override the user input fields, unless you explicitly set override to true.

Example overriding the system field for Anthropic:

# Anthropic doesn't support a system chat type
defaults:
- field: "system"
  value: "answer all questions in french"

Example setting the temperature and overriding max_tokens:

defaults:
- field: "temperature"
  value: 0.5
- field: "max_tokens"
  value: 100

"field": string
"value": .google.protobuf.Value
"override": bool

Postgres

Configuration settings for a Postgres datastore.

"connectionString": string
"collectionName": string

Embedding

Configuration of the API used to generate the embedding.

"openai": .ai.options.gloo.solo.io.Embedding.OpenAI
"azureOpenai": .ai.options.gloo.solo.io.Embedding.AzureOpenAI

OpenAI

Embedding settings for the OpenAI provider.

"authToken": .ai.options.gloo.solo.io.SingleAuthToken

AzureOpenAI

Embedding settings for the Azure OpenAI provider.

"authToken": .ai.options.gloo.solo.io.SingleAuthToken
"apiVersion": string
"endpoint": string
"deploymentName": string

SemanticCache

Cache previous model responses to provide faster responses to similar requests in the future. Results might vary depending on the embedding mechanism used, as well as the similarity threshold set. Semantic caching reduces the number of requests to the LLM provider, improves the response time, and reduces costs.

Example configuring a route to use a redis datastore and OpenAI for RAG:

semanticCache:
  datastore:
    redis:
      connectionString: redis://172.17.0.1:6379
  embedding:
    openai:
      authToken:
        secretRef:
          name: openai-secret
          namespace: gloo-system

"datastore": .ai.options.gloo.solo.io.SemanticCache.DataStore
"embedding": .ai.options.gloo.solo.io.Embedding
"ttl": int
"mode": .ai.options.gloo.solo.io.SemanticCache.Mode

Redis

Settings for a Redis database.

"connectionString": string
"scoreThreshold": float

Weaviate

Settings for a Weaviate database.

"host": string
"httpPort": int
"grpcPort": int
"insecure": bool

DataStore

Data store from which to cache the request and response pairs.

"redis": .ai.options.gloo.solo.io.SemanticCache.Redis
"weaviate": .ai.options.gloo.solo.io.SemanticCache.Weaviate

Mode

The caching mode to use for the request and response lifecycle.

RAG

Retrieval augmented generation (RAG) is a technique of providing relevant context by retrieving relevant data from one or more context datasets and augmenting the prompt with the retrieved information. This can be used to improve the quality of the generated text.

Example configuring a route to use a postgres datastore and OpenAI for RAG:

rag:
  datastore:
    postgres:
      connectionString: postgresql+psycopg://gloo:gloo@172.17.0.1:6024/gloo
      collectionName: default
  embedding:
    openai:
      authToken:
        secretRef:
          name: openai-secret
          namespace: gloo-system

"datastore": .ai.options.gloo.solo.io.RAG.DataStore
"embedding": .ai.options.gloo.solo.io.Embedding
"promptTemplate": string

DataStore

"postgres": .ai.options.gloo.solo.io.Postgres

AIPromptEnrichment

Enrich requests sent to the LLM provider by appending and prepending system prompts. This can be configured only for LLM providers that use the CHAT API type.

Prompt enrichment allows you to add additional context to the prompt before sending it to the model. Unlike RAG or other dynamic context methods, prompt enrichment is static and is applied to every request.

Note: Some providers, including Anthropic, do not support SYSTEM role messages, and instead have a dedicated system field in the input JSON. In this case, use the defaults setting to set the system field.

The following example prepends a system prompt of Answer all questions in French. and appends Describe the painting as if you were a famous art critic from the 17th century. to each request that is sent to the openai HTTPRoute.

apiVersion: gateway.solo.io/v1
kind: RouteOption
metadata:
  name: openai-opt
  namespace: gloo-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: openai
  options:
    ai:
      promptEnrichment:
        prepend:
        - role: SYSTEM
          content: "Answer all questions in French."
        append:
        - role: USER
          content: "Describe the painting as if you were a famous art critic from the 17th century."

"prepend": []ai.options.gloo.solo.io.AIPromptEnrichment.Message
"append": []ai.options.gloo.solo.io.AIPromptEnrichment.Message

Message

An entry for a message to prepend or append to each prompt.

"role": string
"content": string

AIPromptGuard

Set up prompt guards to block unwanted requests to the LLM provider and mask sensitive data. Prompt guards can be used to reject requests based on the content of the prompt, as well as mask responses based on the content of the response.

This example rejects any request prompts that contain the string “credit card”, and masks any credit card numbers in the response.

promptGuard:
  request:
    customResponse: 
      message: "Rejected due to inappropriate content"
    regex:
      action: REJECT
      matches:
      - pattern: "credit card"
        name: "CC"
  response:
    regex:
      builtins:
      - CREDIT_CARD
      action: MASK

"request": .ai.options.gloo.solo.io.AIPromptGuard.Request
"response": .ai.options.gloo.solo.io.AIPromptGuard.Response

Regex

Regular expression (regex) matching for prompt guards and data masking.

"matches": []ai.options.gloo.solo.io.AIPromptGuard.Regex.RegexMatch
"builtins": []ai.options.gloo.solo.io.AIPromptGuard.Regex.BuiltIn
"action": .ai.options.gloo.solo.io.AIPromptGuard.Regex.Action

RegexMatch

Regular expression (regex) matching for prompt guards and data masking.

"pattern": string
"name": string

BuiltIn

Built-in regex patterns for specific types of strings in prompts. For example, if you specify CREDIT_CARD, any credit card numbers in the request or response are matched.

Action

The action to take if a regex pattern is matched in a request or response. This setting applies only to request matches. Response matches are always masked by default.

Webhook

Configure a webhook to forward requests or responses to for prompt guarding.

"host": string
"port": int
"forwardHeaders": []ai.options.gloo.solo.io.AIPromptGuard.Webhook.HeaderMatch

HeaderMatch

Describes how to match a given string in HTTP headers. Match is case-sensitive.

"key": string
"matchType": .ai.options.gloo.solo.io.AIPromptGuard.Webhook.HeaderMatch.MatchType

MatchType

The header string match type.

Moderation

Pass prompt data through an external moderation model endpoint, which compares the request prompt input to predefined content rules. Any requests that are routed through Gloo AI Gateway pass through the moderation model that you specify. If the content is identified as harmful according to the model’s content rules, the request is automatically rejected.

You can configure an moderation endpoint either as a standalone prompt guard setting or in addition to other request and response guard settings.

"openai": .ai.options.gloo.solo.io.AIPromptGuard.Moderation.OpenAI

OpenAI

Configure an OpenAI moderation endpoint.

"model": string
"authToken": .ai.options.gloo.solo.io.SingleAuthToken

Request

Prompt guards to apply to requests sent by the client.

"customResponse": .ai.options.gloo.solo.io.AIPromptGuard.Request.CustomResponse
"regex": .ai.options.gloo.solo.io.AIPromptGuard.Regex
"webhook": .ai.options.gloo.solo.io.AIPromptGuard.Webhook
"moderation": .ai.options.gloo.solo.io.AIPromptGuard.Moderation

CustomResponse

A custom response to return to the client if request content is matched against a regex pattern and the action is REJECT.

"message": string
"statusCode": int

Response

Prompt guards to apply to responses returned by the LLM provider.

"regex": .ai.options.gloo.solo.io.AIPromptGuard.Regex
"webhook": .ai.options.gloo.solo.io.AIPromptGuard.Webhook