LlamaIndex
Use LlamaIndex to point LLM clients to the Gloo AI Gateway.
LlamaIndex is a popular framework for building AI applications and agents. This guide shows you how to use LlamaIndex to point various LLM clients to the Gloo AI Gateway endpoint, rather than directly to the AI provider. Then, Gloo AI Gateway manages authentication and routing to the AI provider. This way, you can take advantage of the advanced features that Gloo AI Gateway provides, such as rate limiting, observability, and more.
Before you begin
Complete the Authenticate with API keys tutorial.
OpenAI
Install the Python dependencies.
pip install llama-index
Save the following Python script to a file. The URL points to the Gloo AI Gateway endpoint by using the route that you defined in the Authenticate with API keys tutorial. Note that this script specifies a fake API key, because the AI Gateway handles authentication.
cat >ai-gateway-llamaindex.py <<EOF import os from llama_index.llms.openai import OpenAI client = OpenAI( api_base="http://" + os.getenv("INGRESS_GW_ADDRESS", "") + ":8080/openai", api_key="fake", ) response = client.complete("Paul Graham is ") print(response) EOF
Run the Python script.
python ai-gateway-llamaindex.py