To configure authentication, you provide the authentication details in the Upstream resource that is backed by the LLM provider endpoint. You have three configuration options for auth: an inline token, a secret reference to an API key, or a passthrough token.

Inline token

Provide the token directly in the configuration for the Upstream. This option is the least secure. Only use this option for quick tests such as trying out AI Gateway.

  1. Get the token from your LLM provider, such as an API key to OpenAI.

  2. Provide the token inline in the Upstream configuration.

      kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  labels:
    app: gloo
  name: openai
  namespace: gloo-system
spec:
  ai:
    openai:
      authToken:
        inline: "$TOKEN"
EOF

  3. Create an HTTPRoute resource that routes incoming traffic to the Upstream. The following example sets up a route on the /openai path to the Upstream backend that you previously created. The URLRewrite filter rewrites the path from /openai to the path of the API in the LLM provider that you want to use, /v1/chat/completions.

      kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: openai
  namespace: gloo-system
spec:
  parentRefs:
    - name: ai-gateway
      namespace: gloo-system
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /openai
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplaceFullPath
          replaceFullPath: /v1/chat/completions
    backendRefs:
    - name: openai
      namespace: gloo-system
      group: gloo.solo.io
      kind: Upstream
EOF

  4. Get the external address of the gateway and save it in an environment variable.

      export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-system gloo-proxy-ai-gateway -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS  
  
      kubectl port-forward deployment/gloo-proxy-ai-gateway -n gloo-system 8080:8080

  5. Send a request to the LLM provider API. Verify that the request succeeds and that you get back a response from the chat completion API.

      curl "$INGRESS_GW_ADDRESS:8080/openai" -H content-type:application/json  -d '{
   "model": "gpt-3.5-turbo",
   "messages": [
     {
       "role": "system",
       "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
     },
     {
       "role": "user",
       "content": "Compose a poem that explains the concept of recursion in programming."
     }
   ]
 }' | jq

    Example output:

      {
  "id": "chatcmpl-AEHYs2B0XUlEioCduH1meERmMwBGF",
  "object": "chat.completion",
  "created": 1727967462,
  "model": "gpt-3.5-turbo-0125",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "In the world of code, a method elegant and rare,\nKnown as recursion, a loop beyond compare.\nLike a mirror reflecting its own reflection,\nIt calls upon itself with deep introspection.\n\nA function that calls itself with artful grace,\nDividing a problem into a smaller space.\nLike a nesting doll, layers deep and profound,\nIt solves complex tasks, looping around.\n\nWith each recursive call, a step is taken,\nTowards solving the problem, not forsaken.\nA dance of self-replication, a mesmerizing sight,\nUnraveling complexity with power and might.\n\nBut beware of infinite loops, a perilous dance,\nWithout a base case, it’s a risky chance.\nFor recursion is a waltz with a delicate balance,\nInfinite beauty, yet a risky dalliance.\n\nSo embrace the concept, in programming’s domain,\nLet recursion guide you, like a poetic refrain.\nA magical loop, a recursive song,\nIn the symphony of code, where brilliance belongs.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 39,
    "completion_tokens": 200,
    "total_tokens": 239,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0
    }
  },
  "system_fingerprint": null
}

API key in a secret

Store the API key in a Kubernetes secret. Then, refer to the secret in the Upstream configuration. This option is more secure than an inline token, because the API key is encoded and you can restrict access to secrets through RBAC rules. Like the inline option, the API key and secret are fairly simple to create and set up. You might use this option in proofs of concept, controlled development and staging environments, or well-controlled prod environments that use secrets.

For steps, see the Auth tutorial.

Passthrough token

Pass through an existing token directly from the client or a successful OpenID Connect (OIDC) connect flow before the request is sent to the Upstream. This option is useful for environments where you set up federated identity for backend clients so that they are already authenticated to the LLM providers that you create Upstreams for. Currently, the request must place the token in the Authorization header.

  1. Make sure that your client is set up as follows:

    • The client that sends a request to the Upstream can authenticate to the LLM provider, such as through an OIDC flow.
    • The authenticated token is sent in requests to the Upstream in an Authentication header.

  2. Configure the Upstream to use passthrough auth.

      kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  labels:
    app: gloo
  name: openai
  namespace: gloo-system
spec:
  ai:
    openai:
      authToken:
        passthrough: {}
EOF

  3. Create an HTTPRoute resource that routes incoming traffic to the Upstream. The following example sets up a route on the /openai path to the Upstream backend that you previously created. The URLRewrite filter rewrites the path from /openai to the path of the API in the LLM provider that you want to use, /v1/chat/completions.

      kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: openai
  namespace: gloo-system
spec:
  parentRefs:
    - name: ai-gateway
      namespace: gloo-system
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /openai
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplaceFullPath
          replaceFullPath: /v1/chat/completions
    backendRefs:
    - name: openai
      namespace: gloo-system
      group: gloo.solo.io
      kind: Upstream
EOF

  4. Trigger your authenticated client to send a request to the Upstream, and verify that you get back a successful response. For example, you might instruct your client to send a curl request through the AI Gateway.

      curl "$INGRESS_GW_ADDRESS:8080/openai" -H content-type:application/json  -d '{
   "model": "gpt-3.5-turbo",
   "messages": [
     {
       "role": "system",
       "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
     },
     {
       "role": "user",
       "content": "Compose a poem that explains the concept of recursion in programming."
     }
   ]
 }' | jq

    Example output:

      {
  "id": "chatcmpl-AEHYs2B0XUlEioCduH1meERmMwBGF",
  "object": "chat.completion",
  "created": 1727967462,
  "model": "gpt-3.5-turbo-0125",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "In the world of code, a method elegant and rare,\nKnown as recursion, a loop beyond compare.\nLike a mirror reflecting its own reflection,\nIt calls upon itself with deep introspection.\n\nA function that calls itself with artful grace,\nDividing a problem into a smaller space.\nLike a nesting doll, layers deep and profound,\nIt solves complex tasks, looping around.\n\nWith each recursive call, a step is taken,\nTowards solving the problem, not forsaken.\nA dance of self-replication, a mesmerizing sight,\nUnraveling complexity with power and might.\n\nBut beware of infinite loops, a perilous dance,\nWithout a base case, it’s a risky chance.\nFor recursion is a waltz with a delicate balance,\nInfinite beauty, yet a risky dalliance.\n\nSo embrace the concept, in programming’s domain,\nLet recursion guide you, like a poetic refrain.\nA magical loop, a recursive song,\nIn the symphony of code, where brilliance belongs.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 39,
    "completion_tokens": 200,
    "total_tokens": 239,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0
    }
  },
  "system_fingerprint": null
}

Cleanup

You can optionally remove the resources that you set up as part of this guide. The following steps assume that you followed you the guides to create API credentials and routing resources for OpenAI.

  kubectl delete secret -n gloo-system openai-secret
kubectl delete upstream -n gloo-system openai
kubectl delete httproute -n gloo-system openai