LlamaIndex is a popular framework for building AI applications and agents. This guide shows you how to use LlamaIndex to point various LLM clients to the Gloo AI Gateway endpoint, rather than directly to the AI provider. Then, Gloo AI Gateway manages authentication and routing to the AI provider. This way, you can take advantage of the advanced features that Gloo AI Gateway provides, such as rate limiting, observability, and more.

Before you begin

Complete the Authenticate with API keys tutorial.

OpenAI

  1. Install the Python dependencies.

      pip install llama-index
      
  2. Save the following Python script to a file. The URL points to the Gloo AI Gateway endpoint by using the route that you defined in the Authenticate with API keys tutorial. Note that this script specifies a fake API key, because the AI Gateway handles authentication.

      cat >ai-gateway-llamaindex.py <<EOF
    import os
    from llama_index.llms.openai import OpenAI
    
    client = OpenAI(
      api_base="http://" + os.getenv("INGRESS_GW_ADDRESS", "") + ":8080/openai",
      api_key="fake",
    )
    response = client.complete("Paul Graham is ")
    print(response)
    EOF
      
  3. Run the Python script.

      python ai-gateway-llamaindex.py