On this page

View traces

Use a custom ConfigMap to configure your Solo Enterprise for agentgateway proxy for tracing.

Before you begin

Install the Solo Enterprise for agentgateway control plane.

Set up an OpenTelemetry collector

Install an OpenTelemetry collector that the Solo Enterprise for agentgateway proxy can send traces to. Depending on your environment, you can further configure your OpenTelemetry to export these traces to your preferred tracing platform, such as Jaeger.

Install the OTel collector.

  helm upgrade --install opentelemetry-collector-traces opentelemetry-collector \
--repo https://open-telemetry.github.io/opentelemetry-helm-charts \
--version 0.127.2 \
--set mode=deployment \
--set image.repository="otel/opentelemetry-collector-contrib" \
--set command.name="otelcol-contrib" \
--namespace=telemetry \
--create-namespace \
-f -<<EOF
config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  exporters:
    otlp/tempo:
      endpoint: http://tempo.telemetry.svc.cluster.local:4317
      tls:
        insecure: true
    debug:
      verbosity: detailed
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch]
        exporters: [debug, otlp/tempo]
EOF

Verify that the collector is up and running.

  kubectl get pods -n telemetry

Example output:

  NAME                                             READY   STATUS    RESTARTS   AGE
opentelemetry-collector-traces-8f566f445-l82s6   1/1     Running   0          17m

Configure your proxy

Create an GlooGatewayParameters resource with your tracing configuration.

  kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1alpha1
kind: GlooGatewayParameters
metadata:
  name: tracing
  namespace: gloo-system
spec:
  rawConfig:
    config: 
      tracing:
        otlpEndpoint: http://opentelemetry-collector-traces.telemetry.svc.cluster.local:4317 # replace with your telemetry collector endpoint
        otlpProtocol: grpc # replace with your telemetry collector protocol
        randomSampling: "true"
        fields:
          add:
            gen_ai.operation.name: '"chat"'
            gen_ai.system: "llm.provider"
            gen_ai.request.model: "llm.requestModel"
            gen_ai.response.model: "llm.responseModel"
            gen_ai.usage.completion_tokens: "llm.outputTokens"
            gen_ai.usage.prompt_tokens: "llm.inputTokens"
EOF

Create your Solo Enterprise for agentgateway proxy. Make sure to reference the GlooGatewayParameters resource that you created so that your proxy starts with the custom tracing configuration.

  kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway-proxy
  namespace: gloo-system
spec:
  gatewayClassName: gloo-gateway-v2
  infrastructure:
    parametersRef:
      name: tracing
      group: gloo.solo.io  
      kind: GlooGatewayParameters  
  listeners:
  - protocol: HTTP
    port: 80
    name: http
    allowedRoutes:
      namespaces:
        from: All
EOF

Verify that your Solo Enterprise for agentgateway proxy is up and running.

  kubectl get pods -n gloo-system

Example output:

  NAMESPACE            NAME                                 READY   STATUS    RESTARTS   AGE
gloo-system      agentgateway-proxy-8b5dc4874-bl79q         1/1     Running   0          12s

Get the external address of the gateway and save it in an environment variable.

  export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-system agentgateway-proxy -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS

  kubectl port-forward deployment/agentgateway-proxy -n gloo-system 8080:80

Set up access to Gemini

Configure access to an LLM provider such as Gemini and send a sample request. You later use this request to verify your tracing configuration.

Save your Gemini API key as an environment variable. To retrieve your API key, log in to the Google AI Studio and select API Keys.
```
  export GOOGLE_KEY=<your-api-key>
  
```

Create a secret to authenticate to Google.

  kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: google-secret
  namespace: gloo-system
type: Opaque
stringData:
  Authorization: $GOOGLE_KEY
EOF

Send a request to the LLM provider API. Verify that the request succeeds and that you get back a response from the chat completion API.

  curl -vik "$INGRESS_GW_ADDRESS/gemini" -H content-type:application/json  -d '{
  "model": "",
  "messages": [
   {"role": "user", "content": "Explain how AI works in simple terms."}
 ]
}'

  curl -vik "localhost:8080/gemini" -H content-type:application/json  -d '{
  "model": "",
  "messages": [
   {"role": "user", "content": "Explain how AI works in simple terms."}
 ]
}'

Example output:

  {"id":"aGLEaMjbLp6p_uMPopeAoAc",
"choices":
  [{"index":0,"message":{
      "content":"Imagine teaching a dog a trick.  You show it what to do, reward it when it's right, and correct it when it's wrong.  Eventually, the dog learns.\n\nAI is similar.  We \"teach\" computers by showing them lots of examples.  For example, to recognize cats in pictures, we show it thousands of pictures of cats, labeling each one \"cat.\"  The AI learns patterns in these pictures – things like pointy ears, whiskers, and furry bodies – and eventually, it can identify a cat in a new picture it's never seen before.\n\nThis learning process uses math and algorithms (like a secret code of instructions) to find patterns and make predictions.  Some AI is more like a dog learning tricks (learning from examples), and some is more like following a very detailed recipe (following pre-programmed rules).\n\nSo, in short: AI is about teaching computers to learn from data and make decisions or predictions, just like we teach dogs tricks.\n",
      "role":"assistant"
      },
   "finish_reason":"stop"
   }],
 "created":1757700714,
 "model":"gemini-1.5-flash-latest",
 "object":"chat.completion",
 "usage":{
     "prompt_tokens":8,
     "completion_tokens":205,
     "total_tokens":213
     }
}

Verify tracing

Get the logs of the agentgateway proxy. In the CLI output, find the trace.id.

  kubectl logs deploy/agentgateway-proxy -n gloo-system

Example output:

  info	request gateway=gloo-system/agentgateway-proxy listener=http 
route=gloo-system/google endpoint=generativelanguage.googleapis.com:443
src.addr=127.0.0.1:49576 http.method=POST http.host=localhost 
http.path=/gemini http.version=HTTP/1.1 http.status=200 
trace.id=d65e4eeb983e2d964e71e8dc8c405f97 span.id=b836e1b1d51b3e74 
llm.provider=gemini llm.request.model= llm.request.tokens=8 
llm.response.model=gemini-1.5-flash-latest llm.response.tokens=313 duration=3165ms

Get the logs of the collector and search for the trace ID. Verify that you see the additional LLM that you configured initially.

  kubectl logs deploy/opentelemetry-collector-traces -n telemetry

Example output:

  Span #0
 Trace ID       : d65e4eeb983e2d964e71e8dc8c405f97
 Parent ID      : 
 ID             : b836e1b1d51b3e74
 Name           : POST /gemini
 Kind           : Server
 Start time     : 2025-09-24 18:12:58.653868462 +0000 UTC
 End time       : 2025-09-24 18:13:01.821700755 +0000 UTC
 Status code    : Unset
 Status message : 
Attributes:
  -> gateway: Str(gloo-system/agentgateway)
  -> listener: Str(http)
  -> route: Str(gloo-system/google)
  -> endpoint: Str(generativelanguage.googleapis.com:443)
  -> src.addr: Str(127.0.0.1:49576)
  -> http.method: Str(POST)
  -> http.host: Str(localhost)
  -> http.path: Str(/gemini)
  -> http.version: Str(HTTP/1.1)
  -> http.status: Int(200)
  -> trace.id: Str(d65e4eeb983e2d964e71e8dc8c405f97)
  -> span.id: Str(b836e1b1d51b3e74)
  -> llm.provider: Str(gemini)
  -> llm.request.model: Str()
  -> llm.request.tokens: Int(8)
  -> llm.response.model: Str(gemini-1.5-flash-latest)
  -> llm.response.tokens: Int(313)
  -> duration: Str(3165ms)
  -> url.scheme: Str(http)
  -> network.protocol.version: Str(1.1)
  -> gen_ai.operation.name: Str(chat)
  -> gen_ai.system: Str(gemini)

Other tracing configurations

Review common tracing providers configurations that you can use with agentgateway.

  kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1alpha1
kind: GlooGatewayParameters
metadata:
  name: jaeger-tracing-config
  namespace: gloo-system
spec:
  rawConfig:
    config: 
      tracing:
        otlpEndpoint: http://jaeger-collector.jaeger.svc.cluster.local:4317
        otlpProtocol: grpc
        randomSampling: "true"
        fields:
          add:
            gen_ai.operation.name: '"chat"'
            gen_ai.system: "llm.provider"
            gen_ai.request.model: "llm.requestModel"
            gen_ai.response.model: "llm.responseModel"
            gen_ai.usage.completion_tokens: "llm.outputTokens"
            gen_ai.usage.prompt_tokens: "llm.inputTokens"
EOF

  kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1alpha1
kind: GlooGatewayParameters
metadata:
  name: langfuse-tracing-config
  namespace: gloo-system
spec:
  rawConfig:
    config: 
      tracing:
        otlpEndpoint: https://us.cloud.langfuse.com/api/public/otel
        otlpProtocol: http
        headers:
          Authorization: "Basic <base64-encoded-credentials>"
        randomSampling: "true"
        fields:
          add:
            gen_ai.operation.name: '"chat"'
            gen_ai.system: "llm.provider"
            gen_ai.prompt: "llm.prompt"
            gen_ai.completion: 'llm.completion.map(c, {"role":"assistant", "content": c})'
            gen_ai.usage.completion_tokens: "llm.outputTokens"
            gen_ai.usage.prompt_tokens: "llm.inputTokens"
            gen_ai.request: "flatten(llm.params)"
EOF

  kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1alpha1
kind: GlooGatewayParameters
metadata:
  name: phoenix-tracing-config
  namespace: gloo-system
spec:
  rawConfig:
    config: 
      tracing:
        otlpEndpoint: http://localhost:4317
        randomSampling: "true"
        fields:
          add:
            span.name: '"openai.chat"'
            openinference.span.kind: '"LLM"'
            llm.system: "llm.provider"
            llm.input_messages: 'flattenRecursive(llm.prompt.map(c, {"message": c}))'
            llm.output_messages: 'flattenRecursive(llm.completion.map(c, {"role":"assistant", "content": c}))'
            llm.token_count.completion: "llm.outputTokens"
            llm.token_count.prompt: "llm.inputTokens"
            llm.token_count.total: "llm.totalTokens"
EOF

  kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1alpha1
kind: GlooGatewayParameters
metadata:
  name: openllmetry-tracing-config
  namespace: gloo-system
spec:
  rawConfig:
    config: 
      tracing:
        otlpEndpoint: http://localhost:4317
        randomSampling: "true"
        fields:
          add:
            span.name: '"openai.chat"'
            gen_ai.operation.name: '"chat"'
            gen_ai.system: "llm.provider"
            gen_ai.prompt: "flattenRecursive(llm.prompt)"
            gen_ai.completion: 'flattenRecursive(llm.completion.map(c, {"role":"assistant", "content": c}))'
            gen_ai.usage.completion_tokens: "llm.outputTokens"
            gen_ai.usage.prompt_tokens: "llm.inputTokens"
            llm.usage.total_tokens: 'llm.totalTokens'
            llm.is_streaming: "llm.streaming"
EOF

Cleanup

You can remove the resources that you created in this guide.

  kubectl delete gateway agentgateway-proxy -n gloo-system
kubectl delete GlooGatewayParameters tracing -n gloo-system 
helm uninstall opentelemetry-collector-traces -n telemetry
kubectl delete httproute google -n gloo-system
kubectl delete Backend google -n gloo-system
kubectl delete secret google-secret -n gloo-system

Solo Enterprise for kgateway

Gloo Gateway (K8s GW API)

Gloo Edge

View traces

Before you begin link

Set up an OpenTelemetry collector link

Configure your proxy link

Set up access to Gemini link

Verify tracing link

Other tracing configurations link

Cleanup link

Before you begin

Set up an OpenTelemetry collector

Configure your proxy

Set up access to Gemini

Verify tracing

Other tracing configurations

Cleanup