On this page
API usage analytics
Debug issues with collecting and displaying usage analytics for the API products in your developer portal.
To collect usage API analytics, you set up several components, including the Gloo OTel pipeline, Clickhouse as a database storage, and the Prometheus chart for the Grafana dashboard. If you cannot see analytics in the Grafana dashboard, try the following troubleshooting steps.
Make sure that all of the related components are healthy.
- Check the Gloo portal components.
kubectl get all -A -l gloo-mesh=gloo-mesh-portal
- Check the Gloo OTel pipeline, including the telemetry gateway and collectors.
kubectl get all -n gloo-mesh -l app.kubernetes.io/name=telemetryGateway kubectl get all -n gloo-mesh -l app.kubernetes.io/name=telemetryCollector
- Check the Clickhouse database storage.
kubectl get all -n gloo-mesh -l app.kubernetes.io/name=clickhouse
- Check the Prometheus stack, particularly the Grafana instance.
kubectl get all -n monitoring -l app.kubernetes.io/part-of=kube-prometheus-stack
- Check the Gloo portal components.
Check that Clickhouse collects the API usage analytics information that the Grafana dashboard needs to display.
- Enable port-forwarding on the Clickhouse pod.
kubectl port-forward $(kubectl get pods -n gloo-mesh -o name | grep clickhouse) 8123 -n gloo-mesh
- Get the
gloo_api_logs
that are stored in Clickhouse. Make sure to update the password to the value that you used in theclickhouse-auth
secret.curl -G 'http://localhost:8123/' \ --data-urlencode 'query=SELECT * FROM gloo_api_logs' \ --data-urlencode 'user=default' \ --data-urlencode 'password=password'
- If you do not see any logs, try generating some traffic to your APIs. If you still do not see logs, review your Portal analytics setup setup for errors.
- Review the logs for the following portal fields. If some of the fields are not populated, revise your portal setup. For example, you might need to create external auth and rate limiting policies for a usage plan.
user_id
api_id
api_product_id
api_product_name
(optional, depending on if you set a product name)usage_plan
Example output with missingusage_plan
data.
...'request_protocol':'HTTP/1.1','response_flags':'-','upstream_cluster':'outbound|5000||tracks-rest-api.tracks.svc.cluster.local','route_name':'tracks-api-tracks-rt.gloo-mesh-gateways.$CLUSTER_NAME--api-example-com-rt.gloo-mesh-gateways.$CLUSTER_NAME','timestamp':'2023-08-03T13:42:48.400Z','status_code':'200','bytes_received':'0','request_id':'194481fb-14e1-4f51-8968-c0d854ff5fad','usage_plan':'<nil>','correlation_id':'<nil>','request_uri':'/trackapi/tracks','user_id':'user1','upstream_local_address':'10.xx.x.x:36388','log.iostream':'stdout','log.file.path':'/var/log/pods/gloo-mesh-gateways_istio-ingressgateway-1-18-1-78b97d9d8d-vgmmn_e3f28703-b6e8-4cd4-82ec-f6e5d6ad48b6/istio-proxy/0.log','client_address':'10.xxx.x.xx2','time':'2023-08-03T13:42:48.86218944Z','logtag':'F','upstream_transport_failure_reason':'<nil>','api_id':'tracks-v1','requested_server_name':'<nil>','bytes_sent':'30984','x_forwarded_for':'10.xxx.x.xx2','api_product_id':'tracks','user_agent':'curl/7.87.0'}
- Enable port-forwarding on the Clickhouse pod.
Check that the Grafana deployment is set up correctly.
- Verify that Grafana uses Clickhouse as a data source, including the correct password that you set up.Example output:
kubectl describe configmap kube-prometheus-stack-grafana-datasource -n monitoring
... datasources: - isDefault: false jsonData: defaultDatabase: default port: 9000 server: clickhouse.gloo-mesh tlsSkipVerify: true username: default name: ClickHouse secureJsonData: password: password type: grafana-clickhouse-datasource uid: clickhouse-access-logs
- Set up port forwarding on your local machine to access the Grafana dashboard.
kubectl port-forward $(kubectl get pods -n monitoring -o name | grep grafana) 8080:3000 -n monitoring
- Open the Grafana dashboard in your web browser and log in, such as with the default
admin
username andprom-operator
password. - Review the API Dashboard for portal usage.
- If a panel does not have any data, click the panel’s name, then click Edit.
- Review the Query information, making sure that the Data source is set to Clickhouse.
- If the Query is missing configuration, review the JSON file for that section. For example, you can search the JSON file for the
Top API consumers
panel. Then, you can find therawSql
query and add or compare to what is in the API Dashboard.
- Verify that Grafana uses Clickhouse as a data source, including the correct password that you set up.