Debug API usage analytics

Debug issues with collecting and displaying usage analytics for the API products in your developer portal.

To collect usage API analytics, you set up several components, including the Gloo OTel pipeline, Clickhouse as a database storage, and the Prometheus chart for the Grafana dashboard. If you cannot see analytics in the Grafana dashboard, try the following troubleshooting steps.

1. Make sure that all of the related components are healthy.
   1. Check the Gloo portal components.

      kubectl get all -A -l gloo-mesh=gloo-mesh-portal
      

   2. Check the Gloo OTel pipeline, including the telemetry gateway and collectors.

      kubectl get all -n gloo-mesh -l app.kubernetes.io/name=telemetryGateway
      kubectl get all -n gloo-mesh -l app.kubernetes.io/name=telemetryCollector
      

   3. Check the Clickhouse database storage.

      kubectl get all -n gloo-mesh -l app.kubernetes.io/name=clickhouse
      

   4. Check the Prometheus stack, particularly the Grafana instance.

      kubectl get all -n monitoring -l app.kubernetes.io/part-of=kube-prometheus-stack
      

2. Check that Clickhouse collects the API usage analytics information that the Grafana dashboard needs to display.
   1. Enable port-forwarding on the Clickhouse pod.

      kubectl port-forward $(kubectl get pods -n gloo-mesh -o name | grep clickhouse) 8123 -n gloo-mesh
      

   2. Get the gloo_api_logs that are stored in Clickhouse. Make sure to update the password to the value that you used in the clickhouse-auth secret.

      curl -G 'http://localhost:8123/' \
      --data-urlencode 'query=SELECT * FROM gloo_api_logs' \
      --data-urlencode 'user=default' \
      --data-urlencode 'password=password'
      

   3. If you do not see any logs, try generating some traffic to your APIs. If you still do not see logs, review your Portal analytics setup setup for errors.
   4. Review the logs for the following portal fields. If some of the fields are not populated, revise your portal setup. For example, you might need to create external auth and rate limiting policies for a usage plan.
      • user_id
      • api_id
      • api_product_id
      • api_product_name (optional, depending on if you set a product name)
      • usage_plan Example output with missing usage_plan data.

      ...'request_protocol':'HTTP/1.1','response_flags':'-','upstream_cluster':'outbound|5000||tracks-rest-api.tracks.svc.cluster.local','route_name':'tracks-api-tracks-rt.gloo-mesh-gateways.$CLUSTER_NAME--api-example-com-rt.gloo-mesh-gateways.$CLUSTER_NAME','timestamp':'2023-08-03T13:42:48.400Z','status_code':'200','bytes_received':'0','request_id':'194481fb-14e1-4f51-8968-c0d854ff5fad','usage_plan':'<nil>','correlation_id':'<nil>','request_uri':'/trackapi/tracks','user_id':'user1','upstream_local_address':'10.xx.x.x:36388','log.iostream':'stdout','log.file.path':'/var/log/pods/gloo-mesh-gateways_istio-ingressgateway-1-18-1-78b97d9d8d-vgmmn_e3f28703-b6e8-4cd4-82ec-f6e5d6ad48b6/istio-proxy/0.log','client_address':'10.xxx.x.xx2','time':'2023-08-03T13:42:48.86218944Z','logtag':'F','upstream_transport_failure_reason':'<nil>','api_id':'tracks-v1','requested_server_name':'<nil>','bytes_sent':'30984','x_forwarded_for':'10.xxx.x.xx2','api_product_id':'tracks','user_agent':'curl/7.87.0'}
      

3. Check that the Grafana deployment is set up correctly.
   1. Verify that Grafana uses Clickhouse as a data source, including the correct password that you set up.

      kubectl describe configmap kube-prometheus-stack-grafana-datasource -n monitoring
      

      Example output:

      ...
      datasources:
      - isDefault: false
        jsonData:
          defaultDatabase: default
          port: 9000
          server: clickhouse.gloo-mesh
          tlsSkipVerify: true
          username: default
        name: ClickHouse
        secureJsonData:
          password: password
        type: grafana-clickhouse-datasource
        uid: clickhouse-access-logs
      

   2. Set up port forwarding on your local machine to access the Grafana dashboard.

      kubectl port-forward $(kubectl get pods -n monitoring -o name | grep grafana) 8080:3000 -n monitoring
      

   3. Open the Grafana dashboard in your web browser and log in, such as with the default admin username and prom-operator password.
   4. Review the API Dashboard for portal usage.
   5. If a panel does not have any data, click the panel's name, then click Edit.
   6. Review the Query information, making sure that the Data source is set to Clickhouse.
   7. If the Query is missing configuration, review the JSON file for that section. For example, you can search the JSON file for the Top API consumers panel. Then, you can find the rawSql query and add or compare to what is in the API Dashboard.