Scrape Gloo management server metrics

Gloo management server metrics are automatically scraped by the built-in Prometheus and can be viewed by accessing the Prometheus expression browser directly or by opening the Gloo operations dashboard in Grafana. However, these metrics are not available to the Gloo telemetry gateway by default.

You can configure your Gloo OTel pipeline to scrape these metrics from the Gloo management server and make these metrics available to the Gloo telemetry gateway so that you can forward them to third-party services.

1. Add the following configuration to your values file for the Gloo Network installation Helm chart. This configuration sets up a scraping job for the Gloo management server. In addition, regex expressions are used to drop or manipulate metrics. The data is then forwarded to the default prometheus exporter.

   telemetryGatewayCustomization:
     extraReceivers:
       prometheus/gloo-mgmt:
         config:
           scrape_configs:
           - job_name: gloo-mesh-mgmt-server-otel
             honor_labels: true
             kubernetes_sd_configs:
             - namespaces:
                 names:
                 - gloo-mesh
               role: pod
             relabel_configs:
             - action: keep
               regex: gloo-mesh-mgmt-server|gloo-mesh-ui
               source_labels:
               - __meta_kubernetes_pod_label_app
             - action: keep
               regex: true
               source_labels:
               - __meta_kubernetes_pod_annotation_prometheus_io_scrape
             - action: drop
               regex: true
               source_labels:
               - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
             - action: replace
               regex: (https?)
               source_labels:
               - __meta_kubernetes_pod_annotation_prometheus_io_scheme
               target_label: __scheme__
             - action: replace
               regex: (.+)
               source_labels:
               - __meta_kubernetes_pod_annotation_prometheus_io_path
               target_label: __metrics_path__
             - action: replace
               regex: ([^:]+)(?::\d+)?;(\d+)
               replacement: $$1:$$2
               source_labels:
               - __address__
               - __meta_kubernetes_pod_annotation_prometheus_io_port
               target_label: __address__
             - action: labelmap
               regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
               replacement: __param_$$1
             - action: labelmap
               regex: __meta_kubernetes_pod_label_(.+)
             - action: replace
               source_labels:
               - __meta_kubernetes_namespace
               target_label: namespace
             - action: replace
               source_labels:
               - __meta_kubernetes_pod_name
               target_label: pod
             - action: drop
               regex: Pending|Succeeded|Failed|Completed
               source_labels:
               - __meta_kubernetes_pod_phase
     extraPipelines:
       metrics/gloo-mgmt:
         receivers:
         - prometheus/gloo-mgmt # Prometheus scrape config for mgmt-server
         processors:
         - memory_limiter
         - batch
         exporters:
         - prometheus # Prometheus deployed by Gloo.
   

2. Follow the Upgrade instructions in the Gloo Mesh Enterprise documentation to apply the changes in your environment. For multicluster environments, upgrade only the Gloo management server with your updated values file, as no upgrade of the Gloo agent is required.

3. Verify that the configmap for the telemetry gateway is updated with the values that you set in the values file.

   kubectl get configmap gloo-telemetry-gateway-config -n gloo-mesh -o yaml
   

4. Perform a rollout restart of the gateway deployment to force your configmap changes to be applied in the telemetry gateway pods.

   kubectl rollout restart -n gloo-mesh deployment/gloo-telemetry-gateway
   

Gloo management server metrics overview

The following metrics are scraped from the Gloo management server and made available to the Gloo telemetry gateway when you enable the Helm configuration in this guide.