Set up API authentication

  1. Create an API key to access the OpenAI API and save it in an environment variable.

      export OPENAI_API_KEY=<insert your API key>
      
  2. Create a Kubernetes secret to store your OpenAI API key.

      kubectl create secret generic openai-secret -n gloo-system \
     --from-literal="Authorization=Bearer $OPENAI_API_KEY" \
     --dry-run=client -oyaml | kubectl apply -f -
      
  3. Create an Upstream resource that configures an LLM provider that references the API key secret that you created earlier.

      kubectl apply -f- <<EOF
    apiVersion: gloo.solo.io/v1
    kind: Upstream
    metadata:
      labels:
        app: gloo
      name: openai
      namespace: gloo-system
    spec:
      ai:
        openai:
          authToken:
            secretRef:
              name: openai-secret
              namespace: gloo-system
    EOF
      
  4. Create an HTTPRoute resource that routes incoming traffic on the /openai path to the Upstream backend that you created earlier. In the example, the URLRewrite filter rewrites the path from /openai to the path of the API in the LLM provider that you want to use, /v1/chat/completions.

      kubectl apply -f- <<EOF
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
      name: openai
      namespace: gloo-system
    spec:
      parentRefs:
        - name: ai-gateway
          namespace: gloo-system
      rules:
      - matches:
        - path:
            type: PathPrefix
            value: /openai
        filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplaceFullPath
              replaceFullPath: /v1/chat/completions
        backendRefs:
        - name: openai
          namespace: gloo-system
          group: gloo.solo.io
          kind: Upstream
    EOF
      
  5. Get the external address of the gateway and save it in an environment variable.

  6. Send a request to the OpenAI API. Verify that the request succeeds and that you get back a response from the chat completion API. Note that you do not send an API key as part of the request. Instead, the Gloo AI Gateway automatically reads the API key from the Kubernetes secret and adds it as a request header.

      curl -v "$INGRESS_GW_ADDRESS:8080/openai" -H content-type:application/json  -d '{
       "model": "gpt-3.5-turbo",
       "messages": [
         {
           "role": "system",
           "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
         },
         {
           "role": "user",
           "content": "Compose a poem that explains the concept of recursion in programming."
         }
       ]
     }' | jq
      

    Example output:

      {
      "id": "chatcmpl-9Z3f0tJ5x4EjsKYZDsk100Q0KOR2v",
      "object": "chat.completion",
      "created": 1718142578,
      "model": "gpt-3.5-turbo-0125",
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "In the realm of code, there lies a wondrous trick,\nA concept so elegant, it'll make your mind pick.\nRecursion it's called, a loop with a twist,\nA function that calls itself, a dance in the mist.\n\nLike a mirror reflecting its own reflection,\nRecursion calls back to solve a problem's inception.\nIt breaks tasks into smaller pieces with grace,\nSolving them one by one, in a recursive embrace.\n\nJust like a Russian nesting doll so grand,\nEach layer revealing a solution so grand.\nThe base case, the exit, the end of the line,\nCompletes the recursion, oh so divine.\n\nInfinite loops it can cause if not tamed,\nBut with careful design, it can be named.\nA powerful tool in the programmer's hand,\nRecursion, a concept so enchanting and grand."
          },
          "logprobs": null,
          "finish_reason": "stop"
        }
     ],
     "usage": {
        "prompt_tokens": 39,
        "completion_tokens": 169,
        "total_tokens": 208
      },
      "system_fingerprint": null
    }
      

Next

Now that you successfully authenticated with your LLM provider, you can go ahead and set up access control and prompt guards for the LLM.