Skip to content
You are viewing the documentation for Solo Enterprise for Istio, formerly known as Gloo Mesh (OSS APIs).

Debug Istio

Page as Markdown
Diagnose issues with your Istio service mesh.
  1. Check the health of the Istio custom resources that relate to the issue you experience. For example, if you see unexpected routing behavior, you might check the destination rules.
    kubectl get istio-io -A
  2. Check the health of the Istio pods and services in each cluster.
    kubectl get pods,svc -A | grep istio
  3. If configuration changes are not taking effect, check the pilot_meshconfig_validation_status metric to get a fast signal on whether istiod rejected the configuration before digging into logs. The metric returns 0 when the current mesh configuration is valid and 1 when it is invalid.
    kubectl exec -n istio-system deploy/istiod -- \
  curl -s localhost:15014/metrics | grep pilot_meshconfig_validation_status
    If the output shows 1, istiod rejected the configuration. Check the istiod logs in the next step for the rejection reason.
  4. Get the logs for the istiod control plane for the cluster where you notice service mesh issues. You can optionally save the output to a local file so that you can read and analyze it more easily.
    kubectl logs -n istio-system deploy/istiod > istiod-logs.txt
  5. If the issue is with a sidecar-enabled workload, try debugging the Envoy proxy container.
  6. If you installed the Solo UI, you can review metrics that are collected by default for Istio components, such as istiod, ztunnel, and waypoint proxies. When reviewing metrics for the ambient data path, keep the following in mind:
    • Metrics are per proxy, such as a ztunnel, sidecar, or waypoint. The client is any workload that has Istio ambient data plane mode or is sidecar-injected. For example, a metric might show that a specific client is failing to reach the workload endpoints behind that ztunnel.
    • Both the source reporter and the destination reporter in ztunnel emit metrics. When a request fails, only the source reporter reports metrics; the destination reporter does not. In failed requests, fields such as destination_app and destination_cluster appear as unknown. To determine the intended destination of a failed request, enable tracing.
    • Endpoint health is per-ztunnel. The health status of all endpoints that the ztunnel serves is available in metrics for that ztunnel. For more information, check out ztunnel outlier detection.
    • The response_flags field in the metric indicates the type of failure (for example, CONNECT).
  7. In a multicluster ambient mesh setup, you can also use istioctl commands in conjunction with the default metrics to follow a packet across clusters in the ambient data path. For example, the path might follow: client → local ztunnel → east-west gateway → remote ztunnel → server. You can:
    • Run istioctl zc connections on east-west gateways to get a populated table of current connections. For example: istioctl zc connections --node <node-name>.
    • Run istioctl zc endpoints to view remote endpoints (IPs and ports) for multicluster services. For example: istioctl zc endpoints --service my-service --service-namespace default.
    • Run istioctl multicluster check --verbose to get diagnostic details of the connected clusters in the ambient mesh, including east-west gateway endpoints, known peer clusters and remote peer gateway addresses, and service sharing summaries. For example: istioctl multicluster check --verbose --contexts="alpha,beta,gamma".
  8. To verify the CA signing certificate and trust anchor that istiod uses for workload authentication, use the istioctl experimental jwks command to retrieve the JWKS document from the running istiod instance.
    istioctl experimental jwks
  9. If you use Grafana to monitor Istio performance, check out the Grafana performance monitoring dashboard in the Solo Communities of Practice (COP) repository.
    Note that COP tools are provided as helpful starting resources that are maintained by the community. These tools are not guaranteed to work in your environment, and are not part of product SLAs.
  10. Review ways from the community to debug your service mesh:
Multicluster peering