For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.
Vertex AI
Configuration and setup for Google Cloud Vertex AI provider.
Configure Vertex AI as an LLM provider in agentgateway.
Before you begin
Install the Solo Enterprise for agentgateway control plane by following the quick start guide.Set up access to Vertex AI
Set up authentication for Vertex AI. Make sure to have your:
- Google Cloud Project ID
- Project location, such as
us-central1(defaults toglobalif not specified) - API key or service account credentials
Save your Vertex AI API key as an environment variable.
export VERTEX_AI_API_KEY=<insert your API key>Create a Kubernetes secret to store your Vertex AI API key.
kubectl apply -f- <<EOF apiVersion: v1 kind: Secret metadata: name: vertex-ai-secret namespace: agentgateway-system type: Opaque stringData: Authorization: $VERTEX_AI_API_KEY EOFCreate an EnterpriseAgentgatewayBackend resource to configure an LLM provider that references the AI API key secret.
kubectl apply -f- <<EOF apiVersion: enterpriseagentgateway.solo.io/v1alpha1 kind: EnterpriseAgentgatewayBackend metadata: name: vertex-ai namespace: agentgateway-system spec: ai: provider: vertexai: model: gemini-pro projectId: "my-gcp-project" region: "us-central1" policies: auth: secretRef: name: vertex-ai-secret EOFCreate an HTTPRoute resource that routes incoming traffic to the EnterpriseAgentgatewayBackend. The following example sets up a route. Note that Solo Enterprise for agentgateway automatically rewrites the endpoint to the appropriate chat completion endpoint of the LLM provider for you, based on the LLM provider that you set up in the EnterpriseAgentgatewayBackend resource.
kubectl apply -f- <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: vertex-ai namespace: agentgateway-system spec: parentRefs: - name: agentgateway-proxy namespace: agentgateway-system rules: - backendRefs: - name: vertex-ai namespace: agentgateway-system group: enterpriseagentgateway.solo.io kind: EnterpriseAgentgatewayBackend EOFkubectl apply -f- <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: vertex-ai namespace: agentgateway-system spec: parentRefs: - name: agentgateway-proxy namespace: agentgateway-system rules: - matches: - path: type: PathPrefix value: /v1/chat/completions backendRefs: - name: vertex-ai namespace: agentgateway-system group: enterpriseagentgateway.solo.io kind: EnterpriseAgentgatewayBackend EOFkubectl apply -f- <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: vertex-ai namespace: agentgateway-system spec: parentRefs: - name: agentgateway-proxy namespace: agentgateway-system rules: - matches: - path: type: PathPrefix value: /vertex backendRefs: - name: vertex-ai namespace: agentgateway-system group: enterpriseagentgateway.solo.io kind: EnterpriseAgentgatewayBackend EOFSend a request to the LLM provider API along the route that you previously created. Verify that the request succeeds and that you get back a response from the API.
Cloud Provider LoadBalancer:
curl "$INGRESS_GW_ADDRESS/v1/chat/completions" -H content-type:application/json -d '{ "model": "", "messages": [ { "role": "user", "content": "Write me a short poem about Kubernetes and clouds." } ] }' | jqLocalhost:
curl "localhost:8080/v1/chat/completions" -H content-type:application/json -d '{ "model": "", "messages": [ { "role": "user", "content": "Write me a short poem about Kubernetes and clouds." } ] }' | jqCloud Provider LoadBalancer:
curl "$INGRESS_GW_ADDRESS/v1/chat/completions" -H content-type:application/json -d '{ "model": "", "messages": [ { "role": "user", "content": "Write me a short poem about Kubernetes and clouds." } ] }' | jqLocalhost:
curl "localhost:8080/v1/chat/completions" -H content-type:application/json -d '{ "model": "", "messages": [ { "role": "user", "content": "Write me a short poem about Kubernetes and clouds." } ] }' | jqCloud Provider LoadBalancer:
curl "$INGRESS_GW_ADDRESS/vertex" -H content-type:application/json -d '{ "model": "", "messages": [ { "role": "user", "content": "Write me a short poem about Kubernetes and clouds." } ] }' | jqLocalhost:
curl "localhost:8080/vertex" -H content-type:application/json -d '{ "model": "", "messages": [ { "role": "user", "content": "Write me a short poem about Kubernetes and clouds." } ] }' | jqExample output:
{ "id": "chatcmpl-vertex-12345", "object": "chat.completion", "created": 1727967462, "model": "gemini-pro", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "In the cloud, Kubernetes reigns,\nOrchestrating pods with great care,\nContainers float like clouds,\nScaling up and down,\nAutomation everywhere." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 12, "completion_tokens": 28, "total_tokens": 40 } }
Next steps
Send file content by URI
In an OpenAI-compatible Chat Completions request, you can put a gs://, http://, or https:// URI in a file content part’s file_data or file_id. The request to Vertex AI uses fileData.
For HTTP(S) signed URLs, MIME type detection uses the extension before the query string or fragment. If the URI does not have a recognized extension, set filename, mime_type, or content_type on the file object.
Use the following request body with the route that you created.
{
"model": "",
"messages": [
{
"role": "user",
"content": [
{ "type": "text", "text": "Compare these documents." },
{
"type": "file",
"file": {
"file_id": "https://storage.googleapis.com/my-bucket/report.pdf?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Signature=abc"
}
},
{
"type": "file",
"file": {
"filename": "summary.pdf",
"file_data": "https://example.com/download/123"
}
}
]
}
]
}