On this page

LLM (OpenAI)

Route requests to OpenAI’s chat completions API with agentgateway on Kubernetes.

Configure OpenAI as an LLM provider in agentgateway.

info

Don’t have an API key to an LLM provider? You can still try out how LLM traffic works in agentgateway by following the httpbun guide. Httpbun provides a mock LLM API endpoint that is compatible with the OpenAI API for chat completions.

Before you begin

Install the Solo Enterprise for agentgateway control plane.

Set up access to OpenAI

Step 1: Get an API key

The following example uses OpenAI. If you use another AI provider, create an API key for that provider’s AI instead, and be sure to modify the example commands in these tutorials to use your provider’s AI API instead.

Create an API key to access the OpenAI API.

Save the API key in an environment variable.

  export OPENAI_API_KEY=${OPENAI_API_KEY:-<insert your API key>}

Create a Kubernetes secret to store your AI API key.

  kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: openai-secret
  namespace: agentgateway-system
type: Opaque
stringData:
  Authorization: $OPENAI_API_KEY
EOF

Step 2: Create the LLM backend

Create an AgentgatewayBackend resource to configure an LLM provider that references the AI API key secret.

  kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai:
        # Optional: specify a default  model
        model: gpt-3.5-turbo
     # Optional: custom host and port, if needed
     # host: api.openai.com  
     # port: 443
  policies:
    auth:
      secretRef:
        name: openai-secret
EOF

Review the following table to understand this configuration. For more information, see the API reference.

Setting	Description
`ai.provider.openai`	Define the OpenAI provider.
`openai.model`	The OpenAI model to use, such as `gpt-3.5-turbo`.
`policies.auth`	Configure the authentication token for OpenAI API. The example refers to the secret that you previously created.

Step 3: Route to the backend

Create an HTTPRoute resource that routes incoming traffic to the AgentgatewayBackend. The following example sets up a route. Note that Solo Enterprise for agentgateway automatically rewrites the endpoint to the OpenAI /v1/chat/completions endpoint.

  kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
    - backendRefs:
      - name: openai
        namespace: agentgateway-system
        group: agentgateway.dev
        kind: AgentgatewayBackend
EOF

  kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /openai
    backendRefs:
    - name: openai
      namespace: agentgateway-system
      group: agentgateway.dev
      kind: AgentgatewayBackend
EOF

Step 4: Send a request to the LLM

Send a request to the LLM provider API along the route that you previously created. Verify that the request succeeds and that you get back a response from the chat completion API.

Cloud Provider LoadBalancer:

  curl "$INGRESS_GW_ADDRESS/v1/chat/completions" -H content-type:application/json  -d '{
   "model": "",
   "messages": [
     {
       "role": "system",
       "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
     },
     {
       "role": "user",
       "content": "Compose a poem that explains the concept of recursion in programming."
     }
   ]
 }' | jq

Localhost:

  curl "localhost:8080/v1/chat/completions" -H content-type:application/json  -d '{
   "model": "",
   "messages": [
     {
       "role": "system",
       "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
     },
     {
       "role": "user",
       "content": "Compose a poem that explains the concept of recursion in programming."
     }
   ]
 }' | jq

Cloud Provider LoadBalancer:

  curl "$INGRESS_GW_ADDRESS/openai" -H content-type:application/json  -d '{
   "model": "",
   "messages": [
     {
       "role": "system",
       "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
     },
     {
       "role": "user",
       "content": "Compose a poem that explains the concept of recursion in programming."
     }
   ]
 }' | jq

Localhost:

  curl "localhost:8080/openai" -H content-type:application/json  -d '{
   "model": "",
   "messages": [
     {
       "role": "system",
       "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
     },
     {
       "role": "user",
       "content": "Compose a poem that explains the concept of recursion in programming."
     }
   ]
 }' | jq

Example output:

  {
  "id": "chatcmpl-AEHYs2B0XUlEioCduH1meERmMwBGF",
  "object": "chat.completion",
  "created": 1727967462,
  "model": "gpt-3.5-turbo-0125",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "In the world of code, a method elegant and rare,\nKnown as recursion, a loop beyond compare.\nLike a mirror reflecting its own reflection,\nIt calls upon itself with deep introspection.\n\nA function that calls itself with artful grace,\nDividing a problem into a smaller space.\nLike a nesting doll, layers deep and profound,\nIt solves complex tasks, looping around.\n\nWith each recursive call, a step is taken,\nTowards solving the problem, not forsaken.\nA dance of self-replication, a mesmerizing sight,\nUnraveling complexity with power and might.\n\nBut beware of infinite loops, a perilous dance,\nWithout a base case, it's a risky chance.\nFor recursion is a waltz with a delicate balance,\nInfinite beauty, yet a risky dalliance.\n\nSo embrace the concept, in programming's domain,\nLet recursion guide you, like a poetic refrain.\nA magical loop, a recursive song,\nIn the symphony of code, where brilliance belongs.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 39,
    "completion_tokens": 200,
    "total_tokens": 239,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0
    }
  },
  "system_fingerprint": null
}

Next steps

Multiple endpoints

Set up other API endpoints such as embeddings or models.

Prompt guards

Set up prompt guards for your LLM traffic.

LLM observability

View metrics and logs for LLM traffic.

Cleanup

You can remove the resources that you created in this guide.

  kubectl delete AgentgatewayBackend openai -n agentgateway-system
kubectl delete HTTPRoute openai -n agentgateway-system
kubectl delete secret openai-secret -n agentgateway-system

Solo Enterprise for agentgateway

LLM (OpenAI)

Before you begin link

Set up access to OpenAI link

Step 1: Get an API key link

Step 2: Create the LLM backend link

Step 3: Route to the backend link

Step 4: Send a request to the LLM link

Next steps link

Cleanup link

link