Debugging the Gloo Mesh UI

You can use the Gloo Mesh UI and other observability tools to help debug your service mesh environment. But what happens when those observability tools, like Gloo Mesh Graph, go wrong?

If something is broken in the UI, you can check several places. The Gloo Mesh metrics pod observes traffic from the Istio sidecar in your pods to the Gloo Mesh apiserver in the management cluster, as shown in the following figure.

Figure: Model of places to check for Gloo Mesh observability issues.

Figure: Model of places to check for Gloo Mesh observability issues.

Check the Prometheus server metrics

The following example uses the Prometheus server that is deployed with Gloo Mesh by default.

  1. Enable port-forwarding of the Prometheus server deployment.

    kubectl port-forward -n gloo-mesh deploy/prometheus-server 9090
    
  2. Open your local host to the port.

    open http://localhost:9090/targets
    
  3. Check for a green UP state, such as in the following figure.

    Figure: Example of green UP state.

    Figure: Example of green UP state.

  4. Check the following query: http://localhost:9090/graph?g0.expr=istio_requests_total&g0.tab=1&g0.stacked=0&g0.range_input=1h.

  5. If you get an Empty query result from the previous step, confirm that your Gloo Mesh agents are not sending metrics to the management server.

    1. Enable port-forwarding on the gloo-mesh-mgmt-server deployment.
      kubectl port-forward -n gloo-mesh deploy/gloo-mesh-mgmt-server 9091
      
    2. Open your local host to the port.
      open http://localhost:9091/metrics
      
    3. Search for istio_requests_total.
    4. If the search returns no results, debug the Gloo Mesh agents.
  6. If you see data returned, check the Prometheus queries.

    1. Set the verbose level to true in the gloo-mesh-enterprise Helm chart. For more information, see Modifying Helm chart values.
    2. Check the logs of the gloo-mesh-ui pod. The logs resemble the following example.
      {"level":"debug","ts":1650923640.6589358,"logger":"prometheus-source","caller":"prom/range.go:59","msg":"executing query 
      sum(
        increase(
          istio_requests_total{
            workload_id=~\".+.bookinfo-experiments.cluster-1|.+.gloo-mesh.cluster-1|.+.bookinfo.cluster-1|.+.bookinfo-demo.cluster-1|.+.bookinfo.cluster-2|.+.bookinfo-security.cluster-2\",
            destination_workload_id=~\".+..+..+\",
            response_code=~\"[2-3].*\",
            reporter=\"source\",
            # Exclude data from outside the mesh
            workload_id!=\"unknown.unknown.unknown\",
          }[15m]
        )
        ) by (
        workload_id,
        destination_workload_id,
        )
      : "}
      
  7. Review the Prometheus queries for the Gloo Mesh Graph.

    open http://localhost:9090/graph