Gloo management server metrics are automatically scraped by the built-in Prometheus and can be viewed by accessing the Prometheus expression browser directly or by opening the Gloo operations dashboard in Grafana. However, because these metrics are not collected by the metrics/ui pipeline, these metrics are not made available to the Gloo telemetry gateway so that you can forward them to third-party solutions, such as Datadog.

By following the steps in this guide, you can configure your Gloo telemetry pipeline to scrape these metrics from the Gloo management server, and make these metrics available to the Gloo telemetry gateway. For a list of the metrics are scraped from the Gloo management server, see Gloo management server metrics.

  1. Get your current installation Helm values, and save them in a file.

      helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $MGMT_CONTEXT > gloo-values.yaml
open gloo-values.yaml

  2. In your Helm values file, set up an extra receiver on the telemetry gateway that scrapes the metrics from the Gloo management server pod. The following configuration uses regex expressions to drop or manipulate metrics from the Gloo management server and forwards the metrics to the default prometheus exporter.

      
telemetryGatewayCustomization:
  extraReceivers:
    prometheus/gloo-mgmt:
      config:
        scrape_configs:
        - job_name: gloo-mesh-mgmt-server-otel
          honor_labels: true
          kubernetes_sd_configs:
          - namespaces:
              names:
              - gloo-mesh
            role: pod
          relabel_configs:
          - action: keep
            regex: gloo-mesh-mgmt-server|gloo-mesh-ui
            source_labels:
            - __meta_kubernetes_pod_label_app
          - action: keep
            regex: true
            source_labels:
            - __meta_kubernetes_pod_annotation_prometheus_io_scrape
          - action: drop
            regex: true
            source_labels:
            - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
          - action: replace
            regex: (https?)
            source_labels:
            - __meta_kubernetes_pod_annotation_prometheus_io_scheme
            target_label: __scheme__
          - action: replace
            regex: (.+)
            source_labels:
            - __meta_kubernetes_pod_annotation_prometheus_io_path
            target_label: __metrics_path__
          - action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $$1:$$2
            source_labels:
            - __address__
            - __meta_kubernetes_pod_annotation_prometheus_io_port
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
            replacement: __param_$$1
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - action: replace
            source_labels:
            - __meta_kubernetes_namespace
            target_label: namespace
          - action: replace
            source_labels:
            - __meta_kubernetes_pod_name
            target_label: pod
          - action: drop
            regex: Pending|Succeeded|Failed|Completed
            source_labels:
            - __meta_kubernetes_pod_phase
  extraPipelines:
    metrics/gloo-mgmt:
      receivers:
      - prometheus/gloo-mgmt # Prometheus scrape config for mgmt-server
      processors:
      - memory_limiter
      - batch
      exporters:
      - prometheus # Prometheus deployed by Gloo.

  3. Upgrade your installation by using your updated values file.

      helm upgrade gloo-platform gloo-platform/gloo-platform --kube-context $MGMT_CONTEXT \
   --version $GLOO_VERSION \
   --namespace gloo-mesh \
   --values gloo-values.yaml

  4. Verify that the configmap for the telemetry gateway is updated with the values that you set in the values file.

      kubectl get configmap gloo-telemetry-gateway-config -n gloo-mesh -o yaml --kube-context $MGMT_CONTEXT

  5. Perform a rollout restart of the gateway deployment to force your configmap changes to be applied in the telemetry gateway pod.

      kubectl rollout restart -n gloo-mesh deployment/gloo-telemetry-gateway --kube-context $MGMT_CONTEXT