AI backends

Use AgentgatewayBackend resources to route waypoint traffic to AI providers like OpenAI and Anthropic.

An AgentgatewayBackend represents an external AI provider or MCP server as a named backend resource. When Solo Enterprise for agentgateway is deployed as a waypoint, you can reference AgentgatewayBackend resources as backendRefs in an HTTPRoute to do path-based routing to multiple AI providers, with per-backend API key injection and provider-aware configuration.

This guide is the east-west equivalent of the standalone Solo Enterprise for agentgateway LLM provider guides. The key difference is that instead of a Gateway with the enterprise-agentgateway GatewayClass for ingress traffic, you use a ServiceEntry and an agentgateway waypoint so that in-mesh workloads can reach LLM providers without embedding credentials in application code. If you only need to control which workloads can make outbound calls to an external API and the application manages its own credentials, see the agentgateway waypoint egress guide instead.

Create AI provider backends

OpenAI is used as an example in the following steps. For a list of all supported providers, see the LLM provider documentation.

1. Set the workload namespace where your agentgateway waypoint is deployed. All resources in this guide are created in this namespace.

   export NAMESPACE=<ns>

2. Create a Kubernetes secret to store your AI API key.

   kubectl create secret generic openai-secret -n ${NAMESPACE} \
     --from-literal=Authorization="Bearer <OPENAI_API_KEY>"

3. Create an AgentgatewayBackend for each AI provider. Each backend specifies the provider, model, and a secret containing the API key.

   kubectl apply -f - <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayBackend
   metadata:
     name: openai
     namespace: ${NAMESPACE}
   spec:
     ai:
       provider:
         openai:
           model: gpt-4o-mini
     policies:
       auth:
         secretRef:
           name: openai-secret
   EOF

Route traffic by path

Create a ServiceEntry for the LLM backend, and an HTTPRoute that references the backend’s ServiceEntry to allow path-based routing.

1. Create a ServiceEntry for a virtual hostname that your in-mesh workloads can use to reach the LLM backends. You can use any hostname, such as myllm.example.com in the following ServiceEntry. The hostname does not need to resolve to a real address, because the placeholder endpoint (0.0.0.1) and resolution: STATIC tell Istio to register the hostname in the service registry without relying on DNS. When a workload sends traffic to this hostname, ztunnel routes it through the agentgateway waypoint (via the istio.io/use-waypoint label), where the HTTPRoute rules in the next step direct it to the correct AgentgatewayBackend.

   kubectl apply -f - <<EOF
   apiVersion: networking.istio.io/v1
   kind: ServiceEntry
   metadata:
     name: myllm
     namespace: ${NAMESPACE}
     labels:
       istio.io/use-waypoint: agentgateway-waypoint
   spec:
     hosts:
     - myllm.example.com
     location: MESH_EXTERNAL
     ports:
     - number: 80
       name: http
       protocol: HTTP
     resolution: STATIC
     endpoints:
     - address: 0.0.0.1
   EOF

2. Create an HTTPRoute that routes requests by path prefix to each AgentgatewayBackend.

   kubectl apply -f - <<EOF
   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: llm-routes
     namespace: ${NAMESPACE}
   spec:
     parentRefs:
     - group: networking.istio.io
       kind: ServiceEntry
       name: myllm
     rules:
     - matches:
       - path:
           type: PathPrefix
           value: /openai
       backendRefs:
       - name: openai
         group: agentgateway.dev
         kind: AgentgatewayBackend
   EOF

Apply authorization policies alongside backends

Apply an EnterpriseAgentgatewayPolicy to the ServiceEntry to restrict which service accounts are allowed to call each path. The policy denies all traffic by default, so any workload not listed in matchExpressions receives a 403 directly from the waypoint, before the request reaches the LLM provider. This 403 (rather than a 401 from the upstream API) confirms the waypoint is intercepting and enforcing policy on the traffic.

1. Set the service account of your agent workload.

   export AGENT_SERVICE_ACCOUNT=<your_agent_sa>

2. Apply an EnterpriseAgentgatewayPolicy. The following example allows requests only from a single service account in the workload namespace.

   kubectl apply -f - <<EOF
   apiVersion: enterpriseagentgateway.solo.io/v1alpha1
   kind: EnterpriseAgentgatewayPolicy
   metadata:
     name: llm-allow-agents-only
     namespace: ${NAMESPACE}
   spec:
     targetRefs:
     - kind: ServiceEntry
       group: "networking.istio.io"
       name: myllm
     traffic:
       authorization:
         action: Allow
         policy:
           matchExpressions:
           - 'source.identity.namespace == "${NAMESPACE}" && source.identity.serviceAccount == "${AGENT_SERVICE_ACCOUNT}"'
   EOF

3. Deploy an unauthorized test pod that runs under the default service account.

   kubectl run curl-unauth -n ${NAMESPACE} --image=curlimages/curl --restart=Never -- sleep 3600

4. Deploy an authorized test pod that runs under the agent service account.

   kubectl run curl-auth -n ${NAMESPACE} --image=curlimages/curl \
     --restart=Never --overrides="{\"spec\":{\"serviceAccountName\":\"${AGENT_SERVICE_ACCOUNT}\"}}" \
     -- sleep 3600

5. Verify that the waypoint enforces the policy by sending a request from the unauthorized pod. The waypoint returns a 403 before the request reaches the LLM provider.

   kubectl exec -n ${NAMESPACE} curl-unauth -- \
     curl -s -o /dev/null -w "%{http_code}" http://myllm.example.com/openai/v1/models

6. Confirm that the authorized pod can reach the API.

   kubectl exec -n ${NAMESPACE} curl-auth -- \
     curl -s -o /dev/null -w "%{http_code}" http://myllm.example.com/openai/v1/models

MCP backends

AgentgatewayBackend also supports MCP servers as targets. For MCP backend configuration, authentication, and tool-level access control, see the Solo Enterprise for agentgateway MCP documentation.

Cleanup

If you no longer need them, you can remove all resources that you created in this guide.

kubectl delete pod curl-unauth curl-auth -n ${NAMESPACE}
kubectl delete enterpriseagentgatewaypolicy llm-allow-agents-only -n ${NAMESPACE}
kubectl delete httproute llm-routes -n ${NAMESPACE}
kubectl delete serviceentry myllm -n ${NAMESPACE}
kubectl delete agentgatewaybackend openai -n ${NAMESPACE}
kubectl delete secret openai-secret -n ${NAMESPACE}