Skip to content
You are viewing the documentation for Solo Enterprise for Istio, formerly known as Gloo Mesh (OSS APIs).

Debug Istio

Page as Markdown
Diagnose issues with your Istio service mesh.
Gloo provides management tools on top of the open source Istio service mesh. If Istio is in a troubled state, you might notice errors in your Gloo environment. For help using the Istio diagnostic tools, see the Istio documentation.
  1. Check the health of the Istio custom resources that relate to the issue you experience. For example, if you see unexpected routing behavior, you might check the destination rules.
kubectl get istio-io -A --context ${REMOTE_CONTEXT}
  1. Check the health of the Istio pods and services in each cluster.
    kubectl get pods,svc -A --context ${REMOTE_CONTEXT} | grep istio
  2. Get the logs for the istiod control plane for the cluster where you notice service mesh issues. You can optionally save the output to a local file so that you can read and analyze it more easily.
    kubectl logs --context ${REMOTE_CONTEXT} -n istio-system deploy/istiod | > istiod-logs.txt
  3. If the issue is with a sidecar-enabled workload, try debugging the Envoy proxy container.
  4. If you installed the Solo UI, you can review metrics that are collected by default for Istio components, such as istiod, ztunnel, and waypoint proxies. When reviewing metrics for the ambient data path, keep the following in mind:
    • Metrics are per proxy, such as a ztunnel, sidecar, or waypoint. The client is any workload that has Istio ambient data plane mode or is sidecar-injected. For example, a metric might show that a specific client is failing to reach the workload endpoints behind that ztunnel.
    • Both the source reporter and the destination reporter in ztunnel emit metrics. Wh en a request fails, only the source reporter reports metrics; the destination reporter does not. In failed requests, fields such as destination_app and destination_cluster appear as unknown. To determine the intended destination of a failed request, enable tracing.
    • Endpoint health is per-ztunnel. The health status of all endpoints that the ztunnel serves is available in metrics for that ztunnel. For more information, check out ztunnel outlier detection.
    • The response_flags field in the metric indicates the type of failure (for example, CONNECT).
  5. In a multicluster ambient mesh setup, you can also use istioctl commands in conjunction with the default metrics to follow a packet across clusters in the ambient data path. For example, the path might follow: client → local ztunnel → east-west gateway → remote ztunnel → server. You can:
    • Run istioctl zc connections on east-west gateways to get a populated table of current connections. For example: istioctl zc connections --node <node-name>.
    • Run istioctl zc endpoints to view remote endpoints (IPs and ports) for multicluster services. For example: istioctl zc endpoints --service my-service --service-namespace default.
    • Run istioctl multicluster check --verbose to get diagnostic details of the connected clusters in the ambient mesh, including east-west gateway endpoints, known peer clusters and remote peer gateway addresses, and service sharing summaries. For example: istioctl multicluster check --verbose --contexts="alpha,beta,gamma".
  6. To verify the CA signing certificate and trust anchor that istiod uses for workload authentication, use the istioctl experimental jwks command to retrieve the JWKS document from the running istiod instance.
    istioctl experimental jwks
  7. If you use Grafana to monitor Istio performance, check out the Grafana performance monitoring dashboard in the Solo Communities of Practice (COP) repository.
  8. Review ways from the community to debug your service mesh:
Multicluster peering