Skip to content
Latest (currently 2026.10.2) has the newest features, bug fixes, and CVE patches of Solo Enterprise for agentgateway.

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Vertex AI

Page as Markdown

Configuration and setup for Google Cloud Vertex AI provider.

Configure Vertex AI as an LLM provider in agentgateway.

Before you begin

Install the Solo Enterprise for agentgateway control plane by following the quick start guide.

Set up access to Vertex AI

  1. Set up authentication for Vertex AI. Make sure to have your:

    • Google Cloud Project ID
    • Project location, such as us-central1 (defaults to global if not specified)
    • API key or service account credentials
  2. Save your Vertex AI API key as an environment variable.

    export VERTEX_AI_API_KEY=<insert your API key>
  3. Create a Kubernetes secret to store your Vertex AI API key.

    kubectl apply -f- <<EOF
    apiVersion: v1
    kind: Secret
    metadata:
      name: vertex-ai-secret
      namespace: agentgateway-system
    type: Opaque
    stringData:
      Authorization: $VERTEX_AI_API_KEY
    EOF
  4. Create an EnterpriseAgentgatewayBackend resource to configure an LLM provider that references the AI API key secret.

    kubectl apply -f- <<EOF
    apiVersion: enterpriseagentgateway.solo.io/v1alpha1
    kind: EnterpriseAgentgatewayBackend
    metadata:
      name: vertex-ai
      namespace: agentgateway-system
    spec:
      ai:
        provider:
          vertexai:
            model: gemini-pro
            projectId: "my-gcp-project"
            region: "us-central1"
      policies:
        auth:
          secretRef:
            name: vertex-ai-secret
    EOF
  5. Create an HTTPRoute resource that routes incoming traffic to the EnterpriseAgentgatewayBackend. The following example sets up a route. Note that Solo Enterprise for agentgateway automatically rewrites the endpoint to the appropriate chat completion endpoint of the LLM provider for you, based on the LLM provider that you set up in the EnterpriseAgentgatewayBackend resource.

    kubectl apply -f- <<EOF
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
      name: vertex-ai
      namespace: agentgateway-system
    spec:
      parentRefs:
        - name: agentgateway-proxy
          namespace: agentgateway-system
      rules:
      - backendRefs:
        - name: vertex-ai
          namespace: agentgateway-system
          group: enterpriseagentgateway.solo.io
          kind: EnterpriseAgentgatewayBackend
    EOF
    kubectl apply -f- <<EOF
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
      name: vertex-ai
      namespace: agentgateway-system
    spec:
      parentRefs:
        - name: agentgateway-proxy
          namespace: agentgateway-system
      rules:
      - matches:
        - path:
            type: PathPrefix
            value: /v1/chat/completions
        backendRefs:
        - name: vertex-ai
          namespace: agentgateway-system
          group: enterpriseagentgateway.solo.io
          kind: EnterpriseAgentgatewayBackend
    EOF
    kubectl apply -f- <<EOF
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
      name: vertex-ai
      namespace: agentgateway-system
    spec:
      parentRefs:
        - name: agentgateway-proxy
          namespace: agentgateway-system
      rules:
      - matches:
        - path:
            type: PathPrefix
            value: /vertex
        backendRefs:
        - name: vertex-ai
          namespace: agentgateway-system
          group: enterpriseagentgateway.solo.io
          kind: EnterpriseAgentgatewayBackend
    EOF
  6. Send a request to the LLM provider API along the route that you previously created. Verify that the request succeeds and that you get back a response from the API.

    Cloud Provider LoadBalancer:

    curl "$INGRESS_GW_ADDRESS/v1/chat/completions" -H content-type:application/json  -d '{
       "model": "",
       "messages": [
         {
           "role": "user",
           "content": "Write me a short poem about Kubernetes and clouds."
         }
       ]
     }' | jq

    Localhost:

    curl "localhost:8080/v1/chat/completions" -H content-type:application/json  -d '{
       "model": "",
       "messages": [
         {
           "role": "user",
           "content": "Write me a short poem about Kubernetes and clouds."
         }
       ]
     }' | jq

    Cloud Provider LoadBalancer:

    curl "$INGRESS_GW_ADDRESS/v1/chat/completions" -H content-type:application/json  -d '{
       "model": "",
       "messages": [
         {
           "role": "user",
           "content": "Write me a short poem about Kubernetes and clouds."
         }
       ]
     }' | jq

    Localhost:

    curl "localhost:8080/v1/chat/completions" -H content-type:application/json  -d '{
       "model": "",
       "messages": [
         {
           "role": "user",
           "content": "Write me a short poem about Kubernetes and clouds."
         }
       ]
     }' | jq

    Cloud Provider LoadBalancer:

    curl "$INGRESS_GW_ADDRESS/vertex" -H content-type:application/json  -d '{
       "model": "",
       "messages": [
         {
           "role": "user",
           "content": "Write me a short poem about Kubernetes and clouds."
         }
       ]
     }' | jq

    Localhost:

    curl "localhost:8080/vertex" -H content-type:application/json  -d '{
       "model": "",
       "messages": [
         {
           "role": "user",
           "content": "Write me a short poem about Kubernetes and clouds."
         }
       ]
     }' | jq

    Example output:

    {
      "id": "chatcmpl-vertex-12345",
      "object": "chat.completion",
      "created": 1727967462,
      "model": "gemini-pro",
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "In the cloud, Kubernetes reigns,\nOrchestrating pods with great care,\nContainers float like clouds,\nScaling up and down,\nAutomation everywhere."
          },
          "finish_reason": "stop"
        }
      ],
      "usage": {
        "prompt_tokens": 12,
        "completion_tokens": 28,
        "total_tokens": 40
      }
    }

Next steps

Send file content by URI

In an OpenAI-compatible Chat Completions request, you can put a gs://, http://, or https:// URI in a file content part’s file_data or file_id. The request to Vertex AI uses fileData.

For HTTP(S) signed URLs, MIME type detection uses the extension before the query string or fragment. If the URI does not have a recognized extension, set filename, mime_type, or content_type on the file object.

Use the following request body with the route that you created.

{
  "model": "",
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "Compare these documents." },
        {
          "type": "file",
          "file": {
            "file_id": "https://storage.googleapis.com/my-bucket/report.pdf?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Signature=abc"
          }
        },
        {
          "type": "file",
          "file": {
            "filename": "summary.pdf",
            "file_data": "https://example.com/download/123"
          }
        }
      ]
    }
  ]
}