Add Cilium metrics

If your cluster uses the Cilium CNI image that is provided by Solo, a few Cilium metrics are collected by default and can be accessed by using the expression browser of the built-in Prometheus server.

To collect more metrics, you can enable the filter/cilium processor on the Gloo telemetry collector agent that is built in to the Gloo telemetry pipeline. This processor collects the following metrics:

To customize the Cilium metrics that you want to collect, you can add your own processor to the Gloo telemetry pipeline. For more information, see Customize the Cilium metrics collection.

Add all Cilium and Hubble metrics

Enable the filter/cilium processor in the Gloo telemetry pipeline to collect Cilium-specific metrics. All metrics are exposed on the Gloo telemetry collector agent where they can be scraped by the built-in Prometheus. You can view these metrics by using the Prometheus expression browser.

  1. Review the configuration of the filter/cilium processor that is built in to the Gloo telemetry pipeline. This configuration includes a regex that scrapes all metrics that start with hubble_, cilium_, or ebpf_solo_io_.*.

    filter/cilium:
      metrics:
        include:
          match_type: regexp
          metric_names:
            - hubble_.*
            - cilium_.*
            - ebpf_solo_io_.*
    
  2. Get your current installation Helm values, and save them in a file.

    helm get values gloo-platform -n gloo-mesh -o yaml > gloo-mesh-enterprise-single.yaml
    open gloo-mesh-enterprise-single.yaml
    
  3. Delete the first line that contains USER-SUPPLIED VALUES:, and save the file.

  4. In your Helm values file, enable the metrics/cilium pipeline that uses the processor that you reviewed in the previous step.

    telemetryCollector:
      enabled: true
    telemetryCollectorCustomization:
      pipelines:
        metrics/cilium:
          enabled: true
    
  5. Upgrade your installation by using your updated values file.

    helm upgrade gloo-platform gloo-platform/gloo-platform \
     --namespace gloo-mesh \
     -f gloo-mesh-enterprise-single.yaml \
     --version $UPGRADE_VERSION
    
  6. Verify that your custom Cilium settings were added to the Gloo telemetry collector configmap.

    kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml
    
  7. Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pod.

    kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent
    
  8. Optional: To monitor your Cilium CNI, import the Cilium dashboard in Grafana.

  1. Review the configuration of the filter/cilium processor that is built in to the Gloo telemetry pipeline. This configuration includes a regex that scrapes all metrics that start with hubble_, cilium_, or ebpf_solo_io_.*.

    filter/cilium:
      metrics:
        include:
          match_type: regexp
          metric_names:
            - hubble_.*
            - cilium_.*
            - ebpf_solo_io_.*
    
  2. Get the Helm values files for your workload cluster.

    helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $REMOTE_CONTEXT > agent.yaml
    open agent.yaml
    
  3. In your Helm values file, enable the metrics/cilium pipeline that uses the processor that you reviewed in the previous step.

    telemetryCollector:
      enabled: true
    telemetryCollectorCustomization:
      pipelines:
        metrics/cilium:
          enabled: true
    
  4. Upgrade the workload cluster.

    helm upgrade gloo-platform gloo-platform/gloo-platform \
      --kube-context $REMOTE_CONTEXT \
      --namespace gloo-mesh \
      -f agent.yaml \
      --version $UPGRADE_VERSION
    
  5. Verify that your settings are applied in the workload cluster.

    1. Verify that the tracing settings were added to the Gloo telemetry collector configmap.

      kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context $REMOTE_CONTEXT
      
    2. Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pods.

      kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context $REMOTE_CONTEXT
      
  6. Optional: To monitor your Cilium CNI, import the Cilium dashboard in Grafana.

Customize the Cilium metrics collection

Instead of enabling all Hubble and Cilium metrics that the Cilium agent emits, you can customize the Cilium processor and include only the metrics that you want to collect. All metrics are exposed on the Gloo telemetry collector agent where they can be scraped by the built-in Prometheus. You can view these metrics by using the Prometheus expression browser.

  1. Get the Cilium and Hubble metrics that you want to collect. For an overview of supported metrics, see the Cilium documentation.

  2. Get your current installation Helm values, and save them in a file.

    helm get values gloo-platform -n gloo-mesh -o yaml > gloo-mesh-enterprise-single.yaml
    open gloo-mesh-enterprise-single.yaml
    
  3. Delete the first line that contains USER-SUPPLIED VALUES:, and save the file.

  4. In your Helm values file, disable the metrics/cilium pipeline, and create a custom pipeline for the metrics that you want to collect. In the following example, the cilium_node_connectivity_status and cilium_node_connectivity_latency_seconds metrics are collected. Then, add your custom processor to the metrics/cilium pipeline.

    telemetryCollector:
      enabled: true
    telemetryCollectorCustomization:
      extraProcessors:
        filter/cilium-custom:
          metrics:
            include:
              match_type: strict
              metric_names:
                # cilium hubble metrics to include
                - cilium_node_connectivity_status
                - cilium_node_connectivity_latency_seconds
      extraPipelines:
        metrics/cilium-custom:
          receivers:
            - prometheus
          processors:
            - memory_limiter
            - filter/cilium-custom
            - batch
          exporters:
            - otlp
    
  5. Upgrade your installation by using your updated values file.

    helm upgrade gloo-platform gloo-platform/gloo-platform \
     --namespace gloo-mesh \
     -f gloo-mesh-enterprise-single.yaml \
     --version $UPGRADE_VERSION
    
  6. Verify that your custom Cilium settings were added to the Gloo telemetry collector configmap.

    kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml
    
  7. Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pod.

    kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent
    
  8. Optional: To monitor your Cilium CNI, import the Cilium dashboard in Grafana.

  1. Get the Cilium and Hubble metrics that you want to collect. For an overview of supported metrics, see the Cilium documentation.

  2. Get the Helm values files for your workload cluster.

    helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $REMOTE_CONTEXT > agent.yaml
    open agent.yaml
    
  3. In your Helm values file, disable the metrics/cilium pipeline, and create a custom pipeline for the metrics that you want to collect. In the following example, the cilium_node_connectivity_status and cilium_node_connectivity_latency_seconds metrics are collected. Then, add your custom processor to the metrics/cilium pipeline.

    telemetryCollector:
      enabled: true
    telemetryCollectorCustomization:
      extraProcessors:
        filter/cilium-custom:
          metrics:
            include:
              match_type: strict
              metric_names:
                # cilium hubble metrics to include
                - cilium_node_connectivity_status
                - cilium_node_connectivity_latency_seconds
      extraPipelines:
        metrics/cilium-custom:
          receivers:
            - prometheus
          processors:
            - memory_limiter
            - filter/cilium-custom
            - batch
          exporters:
            - otlp
    
  4. Upgrade the workload cluster.

    helm upgrade gloo-platform gloo-platform/gloo-platform \
      --kube-context $REMOTE_CONTEXT \
      --namespace gloo-mesh \
      -f agent.yaml \
      --version $UPGRADE_VERSION
    
  5. Verify that your settings are applied in the workload cluster.

    1. Verify that the tracing settings were added to the Gloo telemetry collector configmap.

      kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context $REMOTE_CONTEXT
      
    2. Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pods.

      kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context $REMOTE_CONTEXT
      
  6. Optional: To monitor your Cilium CNI, import the Cilium dashboard in Grafana.