Environment

  1. Get your Gloo Mesh Enterprise version that you run in the management cluster.
      meshctl --kubecontext $MGMT_CONTEXT version
  2. Get the version of Kubernetes that you run in each of your clusters.
      kubectl --context $MGMT_CONTEXT version -o yaml
kubectl --context $REMOTE_CONTEXT version -o yaml
  3. Get the Istio version that you run in each of your workload clusters.
      istioctl --context $REMOTE_CONTEXT version
  4. List the infrastructure provider that hosts your environment, such as AWS or GCP.

Setup

Share the installation method that you used to install Gloo Mesh Enterprise, such as Helm, meshctl, or Argo CD, and the configuration that you used during the installation.

Use the following command to extract the Helm manifests that were used for the Gloo Mesh Enterprise installation on the management cluster.
Replace release-name and gloo-mesh-namespace with the values that you used for your installation.

  helm --kube-context $MGMT_CONTEXT get all <release-name> --namespace <gloo-mesh-namespace> > gloo_mgmt_server_helm.yaml

To extract the agent Helm installation configuration use the following command on the workload clusters.

  helm --kube-context $REMOTE_CONTEXT get all <release-name> --namespace <gloo-mesh-namespace> > gloo_agent_helm.yaml

Share the complete command that you used to install Gloo Mesh Enterprise. The following command shows an example command that you might have used.

  meshctl install --profiles gloo-mesh-single,ratelimit,extauth \
  --set common.cluster=<cluster-name> \
  --set licensing.glooMeshLicenseKey=<gloo-mesh-license-key>

Get the Argo CD applications that define the installation of Gloo Mesh Enterprise by using one of the following methods.

  • kubectl:
      kubectl get applications.argoproj.io/<applicationName> -n <namespaceName> -o yaml
  • Argo CD CLI:
      argocd app get <applicationName> -o yaml

Issue

  1. Provide a detailed description of the issue. Make sure to include the following information:

    • If reproducible, steps to reproduce the issue.
    • Number of workload clusters.
    • Federated trust details, such as self-signed certificates or certificates provided by a certificate management provider.
    • High-level diagram of the clusters showing where Gloo Mesh Enterprise components and applications are running.
    • If you use workspaces, details of how the workspaces are configured.
    • Details of the Istio installation and configuration.

  2. Describe the impact of the issue. For example, the issue might block an update or a demo, or cause the loss of data or an entire system.

  3. Export the relevant configuration files that are related to the issue.

    • Gloo resources: Use the following script to dump all Gloo Mesh Enterprise custom resources into a file. Attach the gloo-mesh-configuration.yaml file to your support request.
        for n in $(kubectl --context <cluster context> get crds | grep solo.io | awk '{print $1}'); do kubectl --context <cluster context> get $n --all-namespaces -o yaml >> gloo-mesh-configuration.yaml; echo "---" >> gloo-mesh-configuration.yaml; done
    • Istio resources: Use the following command to create an Istio bug report that you can attach to your support request. Run it against all the impacted workload clusters.
        istioctl --context $REMOTE_CONTEXT bug-report --istio-namespace <istio-control-plane-namespace>

Product-specific details

Gloo Mesh Enterprise

  1. Capture the output of the meshctl --kubecontext $MGMT_CONTEXT check command.
    Typically, the command output indicates any license errors or details of unhealthy agents, such as in the following example.

      🟢 License status

INFO  gloo-mesh Trial license expiration is 02 Nov 32 10:32 GMT
INFO  gloo-network Trial license expiration is 22 Aug 33 09:53 GMT
INFO  gloo-gateway Trial license expiration is 11 Nov 32 21:33 GMT

🟢 CRD version check


🟢 Gloo Platform deployment status

Namespace | Name                           | Ready | Status
gloo-mesh | gloo-mesh-mgmt-server          | 0/0   | Healthy
gloo-mesh | gloo-mesh-redis                | 1/1   | Healthy
gloo-mesh | gloo-mesh-ui                   | 1/1   | Healthy
gloo-mesh | gloo-telemetry-gateway         | 1/1   | Healthy
gloo-mesh | prometheus-server              | 1/1   | Healthy
gloo-mesh | gloo-telemetry-collector-agent | 3/3   | Healthy

🔴 Mgmt server connectivity to workload agents

ERROR   cluster east-mesh is registered but not connected
ERROR   cluster west-mesh is registered but not connected
Cluster   | Registered | Connected Pod
east-mesh | true       | Not connected
west-mesh | true       | Not connected

Connected Pod | Clusters
Not connected | 2
ERROR   Encountered failed checks.

  2. Collect the logs for various components that are impacted by the issue, such as gloo-mesh-mgmt-server, gloo-telemetry-gateway, gloo-mesh-redis, or gloo-mesh-ui. The components vary depending on your Gloo Mesh Enterprise setup and can be found by default in the gloo-mesh namespace.

    Example: Follow the steps below to get the logs for the mgmt-server controller pod.

    1. Optional: Set the log level to debug.

        kubectl --context $MGMT_CONTEXT patch -n <gloo-namespace> deploy/gloo-mesh-mgmt-server --type "json" -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--verbose=true"}]'

    2. Capture the logs when reproducing the issue.

        kubectl --context $MGMT_CONTEXT logs -f deploy/gloo-mesh-mgmt-server -n <gloo-namespace> > gloo.log

    3. Optional: After you capture the logs, reset the log level to info.

        kubectl --context $MGMT_CONTEXT patch -n <gloo-namespace> deploy/gloo-mesh-mgmt-server --type "json" -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--verbose=false"}]'

    4. Repeat these steps for all the Gloo Mesh Enterprise components in the management cluster.

  3. Get the full debug report for each cluster that is impacted by the issue. The following command creates a tgz file that you can attach to your support ticket.

      meshctl --kubecontext $MGMT_CONTEXT debug report
meshctl --kubecontext $REMOTE_CONTEXT debug report

Istio

  1. If you used Helm to install Istio in the workload clusters, get the Helm configuration that you used during your installation.
      helm --kube-context $REMOTE_CONTEXT get all <release-name> --namespace <istio-system-namespace> > istio_installation_helm.yaml