Tracing

Configure distributed tracing for AI artifacts that are deployed through Solo Enterprise for agentregistry.

About tracing

Solo Enterprise for agentregistry collects distributed traces from agents and MCP servers that are deployed to connected runtimes. Traces let you follow a request end-to-end across agent invocations, LLM calls, and MCP tool calls, so you can diagnose latency, debug failures, and understand how your agents behave in production.

By default, Solo Enterprise for agentregistry deploys a built-in OpenTelemetry (OTel) Collector pipeline alongside the registry server. The collector receives OTLP traces from deployed AI artifacts and stores them in an embedded ClickHouse instance. The ClickHouse instance feeds the Tracing view in the registry UI. You can also forward traces to an external OTEL-compatible backend, such as Jaeger or Grafana Tempo, for deeper analysis or long-term retention. To enable trace collection, you must set the telemetryEndpoint field on each Runtime resource to the OTLP endpoint where agents export their traces.

The following diagram shows how traces are emitted and collected in Solo Enterprise for agentregistry.

    graph LR
    subgraph k8s["Kubernetes cluster"]
        subgraph ar_ns["agentregistry-system namespace"]
            OC["OTel Collector\n(telemetry-collector-pod)"] --> CH["ClickHouse"]
            AR["Solo Enterprise for agentregistry"] -->|"Query traces"| CH
        end
        subgraph runtime_ns["Runtime namespace (e.g. kagent)"]
            AG["Deployed agent"] -->|"OTLP/HTTP (port 4318)"| OC
        end
    end
    OC -.->|"Optional: forward traces"| EXT["External OTEL backend\n(Jaeger, Tempo, etc.)"]
    User(["User"]) -->|"View traces"| AR

Before you begin

Complete one of the following quickstart guides to deploy Solo Enterprise for agentregistry and connect it to a runtime with at least one agent deployed.

Solo Enterprise for kagent Deploy agents to a Kubernetes cluster using Solo Enterprise for kagent as the runtime. AWS Bedrock AgentCore Deploy agents to AWS Bedrock AgentCore as the runtime.

Configure the OTEL host in a Runtime

Each Runtime resource has a telemetryEndpoint field that controls where deployed agents send their OTLP traces. When you deploy an agent or MCP server, the registry injects this endpoint into the agent’s environment so the agent SDK knows where to export spans. If the field is not set, agents do not emit traces.

By default, you point the Runtime at the built-in OTel Collector that is deployed alongside the registry server.

Note

The telemetryEndpoint must be reachable from the namespace where agents run. If your external backend is outside the cluster, make sure the endpoint is accessible from the agent or MCP server and that any required firewall rules or service entries are in place.

Get the address of the built-in OTel Collector.
The kagent runtime runs inside the same cluster as the registry server, so you can reach the OTel Collector through its internal Kubernetes service DNS name. No external exposure is needed.
```
kubectl get svc -n agentregistry-system | grep telemetry
```
Example output:
```
agentregistry-enterprise-telemetry-collector   ClusterIP   10.96.0.1   <none>   4317/TCP,4318/TCP   5m
```

Create or update a Runtime resource with the telemetryEndpoint set to the collector’s OTLP/HTTP port (4318).

arctl apply -f- <<EOF
apiVersion: ar.dev/v1alpha1
kind: Runtime
metadata:
  name: kagent
spec:
  type: Kagent
  telemetryEndpoint: http://agentregistry-enterprise-telemetry-collector.agentregistry-system.svc.cluster.local:4318
  config:
    kagentUrl: http://kagent-controller.kagent:8083
    namespace: kagent
EOF

Setting	Description
`telemetryEndpoint`	The OTLP/HTTP endpoint where deployed agents and MCP servers export their traces. Set to the built-in collector to store traces in ClickHouse and view them in the registry UI. Set to an external endpoint to forward traces to your own OTEL backend instead.

Verify that the Runtime is registered.

arctl get runtimes

Example output:

NAME        TYPE
agentcore   BedrockAgentCore
kagent      Kagent

Note

Bedrock AgentCore runtimes run outside the EKS cluster, so the OTel Collector must be exposed via a LoadBalancer. Make sure that you installed Solo Enterprise for agentregistry with --set telemetry.service.type=LoadBalancer as described in the AWS Bedrock AgentCore quickstart guide.

Retrieve the load balancer hostname and save it as an environment variable.

export OTEL_HOST=$(kubectl -n agentregistry-system get svc \
  agentregistry-enterprise-telemetry-collector \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo "OTel endpoint: http://${OTEL_HOST}:4318"

Create or update a Runtime resource with the telemetryEndpoint set to the collector’s OTLP/HTTP port (4318).

arctl apply -f- <<EOF
apiVersion: ar.dev/v1alpha1
kind: Runtime
metadata:
  name: agentcore
spec:
  type: BedrockAgentCore
  telemetryEndpoint: http://${OTEL_HOST}:4318
  config:
    roleArn: $AWS_ROLE_ARN
    externalId: $AWS_EXTERNAL_ID
    region: $AWS_REGION
EOF

Verify that the Runtime is registered.

arctl get runtimes

Example output:

NAME        TYPE
agentcore   BedrockAgentCore
kagent      Kagent

View traces in the UI

Traces are collected automatically each time a request is sent to a deployed agent. No additional configuration is required when you use the default installation.

Send a request to a deployed agent. Open the Instances view, select your agent, and click Open Chat.
Open the Tracing view. You see a list of recent traces from your deployed agents.
Select a trace to see its details, including individual spans for LLM calls, tool invocations, and agent steps.