Langchain is a popular framework for building AI applications and agents. This guide shows you how to use Langchain to point various LLM clients to the Gloo AI Gateway endpoint, rather than directly to the AI provider. Then, Gloo AI Gateway manages authentication and routing to the AI provider. This way, you can take advantage of the advanced features that Gloo AI Gateway provides, such as rate limiting, observability, and more.

Before you begin

Complete the Authenticate with API keys tutorial.

OpenAI

  1. Install the Python dependencies.

    pip install -qU langchain-openai

  2. Save the following Python script to a file. The URL points to the Gloo AI Gateway endpoint by using the route that you defined in the Authenticate with API keys tutorial. Note that this script specifies a fake API key, because the AI Gateway handles authentication.

    cat >ai-gateway-langchain.py <<EOF
import os
from langchain_openai import ChatOpenAI
from pydantic.v1 import SecretStr

llm = ChatOpenAI(
    model="gpt-4o",
    api_key="fake",  # The gateway supplies the token
    base_url="http://" + os.getenv("INGRESS_GW_ADDRESS", "") + ":8080/openai", # Uses your gateway LB IP address
)

messages = [
(
    "system",
    "You are a helpful assistant that translates English to French. Translate the user sentence.",
),
("human", "I love programming."),
]
ai_msg = llm.invoke(messages)
print(ai_msg.content)
EOF

  3. Run the Python script.

    python ai-gateway-langchain.py