OpenShift comes with built-in Prometheus instances that you can use to monitor metrics for your workloads. Instead of using the built-in Prometheus that Gloo Mesh Core provides, you might want to forward the metrics from the telemetry gateway and collector agents to the OpenShift Prometheus to have a single observability layer for all of your workloads in the cluster.

Single cluster

  1. Get the current values of the Helm release for your Gloo Mesh Core installation. Note that your Helm release might have a different name.

      helm get values gloo-platform -n gloo-mesh -o yaml > gloo-single.yaml
    open gloo-single.yaml
      
  2. In your Helm values file, expose the otlp-metrics and metrics ports on the Gloo collector agent. The otlp-metrics port is used to expose the metrics that were collected by the telemetry collector agent from other workloads in the cluster. The metrics port exposes metrics for the Gloo telemetry collector agents themselves.

      
    telemetryCollector:
      enabled: true
      ports:
        otlp-metrics:
          containerPort: 9091
          enabled: true
          protocol: TCP
          servicePort: 9091
        metrics: 
          enabled: true
          containerPort: 8888
          servicePort: 8888
          protocol: TCP
      
  3. Upgrade your Helm release. Change the release name as needed.

      helm upgrade gloo-platform gloo-platform/gloo-platform \
      --namespace gloo-mesh \
      -f gloo-single.yaml \
      --version $GLOO_VERSION
      
  4. Verify that the Gloo telemetry collector deploys successfully.

      kubectl get pods -n gloo-mesh | grep telemetry
      
  5. Verify that the ports are exposed on the telemetry collector service.

      kubectl get services -n gloo-mesh | grep telemetry
      
  6. Create a configmap to enable workload monitoring in the cluster.

      kubectl apply -f - <<EOF
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: cluster-monitoring-config
      namespace: openshift-monitoring
    data:
      config.yaml: |
        enableUserWorkload: true
    EOF
      
  7. Create a service monitor resource to instruct the OpenShift Prometheus to scrape metrics from the Gloo telemetry collector agent. The service monitor scrapes metrics from the otlp-metrics and metrics ports that you exposed earlier.

      kubectl apply -f- <<EOF
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: gloo-telemetry-collector-sm
      namespace: gloo-mesh
    spec:
      endpoints:
      - interval: 30s
        port: otlp-metrics
        scheme: http
      - interval: 30s
        port: metrics
        scheme: http
      selector:
        matchLabels:
          app.kubernetes.io/name: telemetryCollector
    EOF
      
  8. Open the OpenShift web console and select the Administrator view.

  9. Navigate to Observe > Metrics to open the built-in Prometheus expression browser.

  10. Verify that you can see metrics for the telemetrycollector container. For example, you can enter otelcol_exporter_sent_metric_points in the expression browser and verify that these metrics were sent. For an overview of metrics that are exposed, see Default metrics in the pipeline.

Review the Next section for optional steps that might help you use metrics in OpenShift Prometheus.

Multicluster

Management cluster

  1. Get the current values of the Helm release for the management cluster. Note that your Helm release might have a different name.

      helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $MGMT_CONTEXT > mgmt-server.yaml
    open mgmt-server.yaml
      
  2. In your Helm values file for the management cluster, expose the otlp-metrics and metrics ports on the Gloo telemetry gateway and the metrics port of the Gloo telemetry collector agent. The otlp-metrics port is used to expose the metrics that were collected by the telemetry collector agents across workload clusters and sent to the telemetry gateway. The metrics port exposes metrics for the Gloo telemetry gateway and collector agents themselves.

      
    telemetryGateway:
      enabled: true
      service:
        type: LoadBalancer
      ports:
        otlp-metrics:
          containerPort: 9091
          enabled: true
          protocol: TCP
          servicePort: 9091
        metrics: 
          enabled: true
          containerPort: 8888
          servicePort: 8888
          protocol: TCP
    telemetryCollector:
      enabled: true
      ports:
        otlp-metrics:
          containerPort: 9091
          enabled: true
          protocol: TCP
          servicePort: 9091       
        metrics: 
          enabled: true
          containerPort: 8888
          servicePort: 8888
          protocol: TCP
      
  3. Upgrade your Helm release in the management cluster. Change the release name as needed.

      helm upgrade gloo-platform gloo-platform/gloo-platform \
     --kube-context $MGMT_CONTEXT \
     --namespace gloo-mesh \
     -f mgmt-server.yaml \
     --version $GLOO_VERSION
      
  4. Verify that the Gloo telemetry gateway and collector agents deploy successfully.

      kubectl get pods --context $MGMT_CONTEXT -n gloo-mesh | grep telemetry
      
  5. Verify that the ports are exposed on the telemetry collector and gateway services.

      kubectl get services --context $MGMT_CONTEXT -n gloo-mesh | grep telemetry
      
  6. Create a configmap to enable workload monitoring for the management cluster.

      kubectl --context $MGMT_CONTEXT apply -f - <<EOF
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: cluster-monitoring-config
      namespace: openshift-monitoring
    data:
      config.yaml: |
        enableUserWorkload: true
    EOF
      
  7. Create a service monitor resource to instruct the OpenShift Prometheus to scrape metrics from the Gloo telemetry gateway. The service monitor scrapes metrics from the otlp-metrics and metrics ports that you exposed earlier.

      kubectl --context ${MGMT_CONTEXT} apply -f- <<EOF
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: gloo-telemetry-gateway-sm
      namespace: gloo-mesh
    spec:
      endpoints:
      - interval: 30s
        port: otlp-metrics
        scheme: http
      - interval: 30s
        port: metrics
        scheme: http
      selector:
        matchLabels:
          app.kubernetes.io/name: telemetryGateway
    EOF
      
  8. Create another service monitor to scrape metrics from the Gloo telemetry collector agent in the management cluster.

      kubectl --context ${MGMT_CONTEXT} apply -f- <<EOF
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: gloo-telemetry-collector-sm
      namespace: gloo-mesh
    spec:
      endpoints:
      - interval: 30s
        port: metrics
        scheme: http
      - interval: 30s
        port: otlp-metrics
        scheme: http
      selector:
        matchLabels:
          app.kubernetes.io/name: telemetryCollector
    EOF
      
  9. Open the OpenShift web console for the management cluster and select the Administrator view.

  10. Navigate to Observe > Metrics to open the built-in Prometheus expression browser.

  11. Verify that you can see metrics for the telemetrygateway and telemetrycollector containers. For example, you can enter otelcol_exporter_sent_metric_points in the expression browser and verify that these metrics were sent from both containers. For an overview of metrics that these two components expose, see Default metrics in the pipeline.

Workload cluster

  1. Get the current values of the Helm release for the workload cluster. Note that your Helm release might have a different name.

      helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $REMOTE_CONTEXT > data-plane.yaml
    open data-plane.yaml
      
  2. In your Helm values file for the workload cluster, expose the metrics port on the Gloo telemetry collector agent. The metrics port exposes metrics for the Gloo telemetry collector agents, such as otelcol_exporter_enqueue_failed_metric_points, that you can use to determine whether the connection between the collector agents and the telemetry gateway in the management cluster is healthy.

      
    telemetryCollector:
      enabled: true
      ports:
        metrics: 
          enabled: true
          containerPort: 8888
          servicePort: 8888
          protocol: TCP
      
  3. Upgrade your Helm release in each workload cluster. Change the release name as needed. Be sure to update the cluster context for each workload cluster that you repeat this command for.

      helm upgrade gloo-platform gloo-platform/gloo-platform \
      --kube-context $REMOTE_CONTEXT \
      --namespace gloo-mesh \
      -f data-plane.yaml \
      --version $GLOO_VERSION
      
  4. Verify that the Gloo telemetry collector agents deploy successfully.

      kubectl get pods --context $REMOTE_CONTEXT -n gloo-mesh | grep telemetry
      
  5. Verify that the port is exposed on the telemetry collector service.

      kubectl get services --context $REMOTE_CONTEXT -n gloo-mesh | grep telemetry
      
  6. Create a service monitor to scrape metrics from the Gloo telemetry collector agent in the management cluster.

      kubectl --context ${REMOTE_CONTEXT} apply -f- <<EOF
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: gloo-telemetry-collector-sm
      namespace: gloo-mesh
    spec:
      endpoints:
      - interval: 30s
        port: metrics
        scheme: http
      selector:
        matchLabels:
          app.kubernetes.io/name: telemetryCollector
    EOF
      
  7. Create a configmap to enable workload monitoring for the workload cluster.

      kubectl --context $REMOTE_CONTEXT apply -f - <<EOF
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: cluster-monitoring-config
      namespace: openshift-monitoring
    data:
      config.yaml: |
        enableUserWorkload: true
    EOF
      
  8. Create a service monitor to scrape metrics from the istiod service mesh control plane.

      kubectl --context ${REMOTE_CONTEXT1} apply -f- <<EOF
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: istiod-http-monitoring
      namespace: istio-system
    spec:
      endpoints:
      - interval: 30s
        port: http-monitoring
        scheme: http
      selector:
        matchLabels:
          app: istiod
    EOF
      
  9. Open the OpenShift web console for the workload cluster and select the Administrator view.

  10. Navigate to Observe > Metrics to open the built-in Prometheus expression browser.

  11. Verify that you can see metrics for the telemetrycollector containers. For example, you can enter otelcol_exporter_sent_metric_points in the expression browser. For an overview of metrics that are exposed, see Default metrics in the pipeline.

  12. Optional: Repeat these steps for each workload cluster.

Review the Next section for optional steps that might help you use metrics in OpenShift Prometheus.

Next

Now that you set up metrics to flow from your Gloo Mesh Core open telemetry pipeline to the OpenShift Prometheus, review the following options to do more with the metrics.

Gloo UI

You can update the Gloo UI to read metrics from the OpenShift Prometheus instance to populate the Gloo UI graph and other metrics. This way, you can remove the built-in Prometheus instance that Gloo Mesh Core provides.

For more information, see Connect the Gloo UI to OpenShift Prometheus.

Alerts

Create alerts with your OpenShift monitoring tools. The following subsections show configuration files for Gloo, OTel, and Istio alerts.

For steps and more information about other metrics, review the following resources:

Translation alert

You might create an alert to monitor the translation time based on Gloo Mesh Core metrics. The following example Prometheus rule creates an alert when translation time is higher than ten seconds (10).

For complex environments with thousands of services and many clusters, this translation time might be normal. You can adjust this time based on your environment and expected performance. For expected translation and other scalability thresholds, see the Gloo Mesh scalability docs.

Example Prometheus rule:

  kubectl --context ${MGMT_CONTEXT} apply -f- <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gloo-platform-translation-latency-is-high
  namespace: gloo-mesh
spec:
  groups:
  - name: GlooPlatformAlerts
    rules:
    - alert: GlooPlatformTranslationLatencyIsHigh
      for: 15m 
      expr: histogram_quantile(0.99, sum(rate(gloo_mesh_translation_time_sec_bucket[5m])) by(le)) > 10
      labels:
        severity: warning 
      annotations:
        runbook: https://docs.solo.io/gloo-mesh-enterprise/main/troubleshooting/gloo/server-relay/
        summary: The translation time exceeds 10 seconds.
EOF
  

OTel connectivity alert

You might create alerts to check the connectivity between the OTel gateways in the management cluster and the collectors in the workload clusters. For example, if the otelcol_exporter_queue_size metric increases consistently, this might indicate that the collectors can no longer send traffic to the gateway.

The following example sets up a warning if the OTel collector queue size exceeds 100. Adjust this value based on what is a normal queue size for your environment, so that you get alerted when the normal queue is exceeded. The default queue size is 1,000.

For more information about setting up monitoring for queues, see the OpenTelemetry docs.

Example Prometheus rule:

  kubectl --context ${MGMT_CONTEXT} apply -f- <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gloo-platform-otel-queue-is-high
  namespace: gloo-mesh
spec:
  groups:
  - name: GlooPlatformAlerts
    rules:
    - alert: GlooPlatformOTelQueueIsHigh
      for: 15m 
      expr: predict_linear(otelcol_exporter_queue_size[15m], 900) >= 100
      labels:
        severity: warning 
      annotations:
        runbook: https://docs.solo.io/gloo-mesh-enterprise/main/troubleshooting/gloo/telemetry/
        summary: The Gloo OTel queue is building up. If the queue size is increasing consistently, this might indicate network issues. Check the connectivity between the gateway and collectors for any issues.
EOF
  

Istio alert

You might create an alert to monitor the time that it takes for the Istio proxy to get a configuration change. The following example Prometheus rule creates an alert when proxy convergence time is higher than one minute (60 seconds).

For complex environments with thousands of services and many clusters, this translation time might be normal. You can adjust this time based on your environment and expected performance. For expected translation and other scalability thresholds, see the Gloo Mesh scalability docs.

Example Prometheus rule:

  kubectl --context ${MGMT_CONTEXT} apply -f- <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gloo-platform-istio-proxy-convergence-is-high
  namespace: gloo-mesh
spec:
  groups:
  - name: GlooPlatformAlerts
    rules:
    - alert: GlooPlatformIstioProxyConvergenceIsHigh
      for: 15m 
      expr: histogram_quantile(0.99, sum(rate(pilot_proxy_convergence_time_bucket[1m])) by (le)) > 60
      labels:
        severity: warning 
      annotations:
        runbook: https://docs.solo.io/gloo-mesh-enterprise/main/troubleshooting/service-mesh/istio/
        summary: The Istio proxy is taking more than 60 seconds to get configuration changes.
EOF