In this guide we explore how to troubleshoot when things don't behave the way you're expecting. We also try to explain how things work under the covers so you can use your own troubleshooting skills to find why things may not behave as expected. We would love to hear from you if you if you get stuck on the Slack.

Helpful debugging tips

So you've created some TrafficPolicy rules or just grouped your first VirtualMesh and it's not behaving as expected. What do you do?

One of the first places to always start is meshctl check. This command is your friend. Here's what it does:

  1. Tests connectivity to the Kubernetes cluster.
  2. Checks the minimum supported Kubernetes minor version.
  3. Checks that the Gloo Mesh installation's namespace exists, which is gloo-mesh by default.
  4. Verifies that the Gloo Mesh enterprise-networking and enterprise-agent components are installed and running.
  5. Verifies that none of the Destinations have any federation errors.

Checking federation status is likely the most helpful after you've tried to apply a VirtualMesh. Often it's best to check the VirtualMesh CR status field to make sure it doesn't see any issues:

kubectl get virtualmesh -n gloo-mesh -o jsonpath='{.items[].status}'

NOTE: all of the Gloo Mesh CRs have a status field that give you an indication of what has happened with its processing. Gloo Mesh, like most operator-style controllers, implements a state machine and helps resources transition between states. All of the current state transitions are available on this status field of any of the Gloo Mesh CRs.

CRD Statuses

Gloo Mesh's CRD statuses reflect the system's state to facilitate diagnosing configuration issues.

The following discussion will use TrafficPolicy as an example, but the concepts are applicable to all CRD statuses. We encourage users to read the API documentation to familiarize themselves with the status semantics for the various Gloo Mesh objects, as they are generally useful for understanding the state of the system.

Consider the following TrafficPolicy resource:

kind: TrafficPolicy
  generation: 1
  name: mgmt-reviews-outlier
  namespace: gloo-mesh
    - kubeServiceRefs:
        - clusterName: cluster-1
          name: reviews
          namespace: bookinfo
        - clusterName: cluster-2
          name: reviews
          namespace: bookinfo
  observedGeneration: 1
  state: ACCEPTED
      state: ACCEPTED
      state: ACCEPTED
  - istio-ingressgateway-istio-system-cluster-1-deployment.gloo-mesh.
  - istio-ingressgateway-istio-system-cluster-2-deployment.gloo-mesh.
  - productpage-v1-bookinfo-cluster-1-deployment.gloo-mesh.
  - ratings-v1-bookinfo-cluster-1-deployment.gloo-mesh.
  - ratings-v1-bookinfo-cluster-2-deployment.gloo-mesh.
  - reviews-v1-bookinfo-cluster-1-deployment.gloo-mesh.
  - reviews-v2-bookinfo-cluster-1-deployment.gloo-mesh.
  - reviews-v3-bookinfo-cluster-2-deployment.gloo-mesh.

Observed generation:

When diagnosing why networking configuration is not being applied as expected, checking the relevant resource's observedGeneration field is a good first step. An object's metadata.generation field is incremented by the Kubernetes server whenever the object's spec changes. If the status.ObservedGeneration equals metadata.generation, this means that Gloo Mesh has successfully processed the latest version of that configuration resource. If this is not the case, it's usually a sign that an unexpected system error occurred, in which case the next debugging step should be to check the logs of Gloo Mesh pods (detailed below).


The status.state field reports the overall state of the resource. The exact semantics of this field depend on the CRD in question, so check the protobuf documentation for details.


Networking configuration CRDs operate on Destinations, which are discovered by Gloo Mesh's discovery comoponent. The TrafficPolicy CRD selects traffic destinations (spec.destinationSelector) to configure. Configuration can successfully apply to some Destinations but not others. This per-target state is reflected in the status.destinations field.

If a Destination does not appear in this list, it can mean either that the Destination was not selected properly (see the proto documentation for ServiceSelector for detailed semantics), or that the Destination has not been discovered. You can examine discovered Destinations by using meshctl describe destination (this command will also show all networking configuration applied to each Destination).


This field shows all workloads, i.e. traffic origins, that the policy applies to.


Knowing how to get logs from the Gloo Mesh components is crucial for getting feedback about what's happening. Gloo Mesh has three core components:

When troubleshooting various parts of the Gloo Mesh functionality, you will likely want to see the logs for some of these components. For example, when creating a new VirtualMesh and you see something like PROCESSING_ERROR in the federation status, check the logs for the mesh-networking component like this:

kubectl logs -f deploy/networking -n gloo-mesh

This will give you valuable insight into the control loop that manages the state of a VirtualMesh

Debug Logging

You can set the env variable DEBUG_MODE to “true” in any of the Gloo Mesh pods to increase the logging level. You can set an explicit logging level using the LOG_LEVEL env variable. See for the log levels.

When you find unexpected behaviors with your request handling, here are a few areas to look in Envoy that can aid in debugging. The Envoy Admin API can be viewed by port-forwarding the istio-ingressgateway service:

kubectl port-forward -n gloo-mesh deploy/istio-ingressgateway 15000:15000

This way you can visit localhost:15000 and get access to the Envoy Admin API.

Common issues

When following the guides for multi-cluster and multi-mesh communication, the following things could happen of which you should be aware:

Cluster shared identity failed

You can check that the second cluster is reachable, that it has the gloo-mesh namespace, that it has the csr-agent running successfully, and that a VirtualMeshCertificateSigningRequest has been created:

kubectl get virtualmeshcertificatesigningrequest -n gloo-mesh --context cluster-2-context

If everything looks fine, ie the status field is in ACCEPTED state, make sure the Istio cacerts were created in the istio-system-namespace

kubectl get secret -n istio-system cacerts --context cluster-2-context

One thing to NOT overlook is the fact that Istio's control plane, istiod needs to be restarted once the cacerts has been created or changed.

For example:

kubectl --context cluster-2-context delete pod -n istio-system -l app=istiod 

This is being improved in future versions of Istio

Communication across clusters isn't working

For communication in a replicated control plane, shared root-trust Virtual Mesh to work in the absence of a flat network, Istio needs to be installed correctly with its Ingress Gateway. At the moment, either a LoadBalancer or NodePort can be used for the Ingress Gateway service. Either way, the Host for the NodePort or the IP for the LoadBalancer needs to be reachable between clusters.

For example, on a cloud provider like GKE, we may see something like this:

kubectl get svc -n istio-system
NAME                   TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)                                                                                                                                      AGE
istio-ingressgateway   LoadBalancer   15020:32347/TCP,80:32414/TCP,443:32713/TCP,15029:31362/TCP,15030:32242/TCP,15031:31899/TCP,15032:32471/TCP,31400:31570/TCP,15443:30416/TCP   4d2h
istio-pilot            ClusterIP   <none>           15010/TCP,15011/TCP,15012/TCP,8080/TCP,15014/TCP,443/TCP                                                                                     4d2h
istiod                 ClusterIP    <none>           15012/TCP,443/TCP      

Note the external-ip. Any ServiceEntry created for cross-cluster service discovery will use this external ip.

Do cross-cluster service entries resolve DNS?

If you've installed Istio 1.8 or greater with Smart DNS enabled per our Istio install guides, then you're all set. The Istio sidecar agent is dynamically configured by Istiod to resolve host names for all accessible Kubernetes services and Istio service entries. The agent will then transparently intercept and resolve DNS queries.

If not, you can manually set up the DNS yourself. See the customizing DNS for Istio routing for more.

What if I'm managing Istio installations across clusters on a flat network?

In a flat network environment, where pod IPs in one cluster are directly accessible via pods in another, you may wish to bypass the ingress gateways that typically mediate cross-cluster routing in a multi-primary, multi-network deployment. Gloo Mesh Enterprise users can leverage the flat network flag on the Virtual Mesh API to configure Gloo Mesh to federate services such that remote services are accessed directly without first passing through an ingress gateway.

What Istio versions are supported?

Right now, Gloo Mesh supports Istio 1.7.x, 1.8.x and 1.9.x.

Istio versions 1.8.0, 1.8.1, and 1.8.2 have a known issue where sidecar proxies may fail to start under specific circumstances. This bug may surface in sidecars configured for static failover by Virtual Destinations. This issue is resolved in Istio 1.8.3.

Found something else?

If you've run into something else, please reach out in the #gloo-mesh channel on the Slack or file an issue on GitHub.