Add Cilium metrics
Monitor your Cilium CNI by enabling metrics for the Cilium agent, operator, and Hubble.
Overview
If your cluster uses the Cilium CNI image that is provided by Solo, a few Cilium metrics are collected by default and can be accessed by using the expression browser of the built-in Prometheus server.
To collect more metrics, you can enable the filter/cilium
processor on the Gloo telemetry collector agent that is built in to the Gloo telemetry pipeline. This processor collects the following metrics:
- Cilium and Hubble: By default, all Cilium and Hubble metrics are collected. For an overview of metrics that are collected, see the Cilium documentation.
To customize the Cilium metrics that you want to collect, you can add your own processor to the Gloo telemetry pipeline. For more information, see Customize the Cilium metrics collection.
Add all Cilium and Hubble metrics
Enable the filter/cilium
processor in the Gloo telemetry pipeline to collect Cilium-specific metrics. All metrics are exposed on the Gloo telemetry collector agent where they can be scraped by the built-in Prometheus. You can view these metrics by using the Prometheus expression browser.
Single cluster
Check your Cilium CNI Helm values.
helm get values cilium -n kube-system -o yaml > cilium.yaml open cilium.yaml
- If
customCalls.enabled
is set totrue
, continue to the next steps. - If
customCalls.enabled
is set tofalse
or is unset, perform a Helm upgrade to set it totrue
. This setting is required to collect the additional eBPF metrics.
- If
Review the configuration of the
filter/cilium
processor that is built in to the Gloo telemetry pipeline. This configuration includes a regex that scrapes all metrics that start withhubble_
orcilium_
.filter/cilium: metrics: include: match_type: regexp metric_names: - hubble_.* - cilium_.*
Get your current installation Helm values, and save them in a file.
helm get values gloo-platform -n gloo-mesh -o yaml > gloo-single.yaml open gloo-single.yaml
In your Helm values file, enable the
metrics/cilium
pipeline that uses the processor that you reviewed in the previous step. Be sure thatglooNetwork.enabled
is also included in your values.glooNetwork: enabled: true telemetryCollector: enabled: true telemetryCollectorCustomization: pipelines: metrics/cilium: enabled: true
Upgrade your installation by using your updated values file.
helm upgrade gloo-platform gloo-platform/gloo-platform \ --namespace gloo-mesh \ --version $GLOO_VERSION \ --values gloo-single.yaml
Verify that your settings were added to the Gloo telemetry collector configmap.
kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml
Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pod.
kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent
Optional: To monitor your Cilium CNI, import the Cilium dashboard in Grafana.
Multicluster
Check the Cilium CNI Helm values on a workload cluster.
helm get values cilium -n kube-system --kube-context $REMOTE_CONTEXT -o yaml > cilium.yaml open cilium.yaml
- If
customCalls.enabled
is set totrue
, continue to the next steps. - If
customCalls.enabled
is set tofalse
or is unset, perform a Helm upgrade to set it totrue
. This setting is required to collect the additional eBPF metrics.
- If
Review the configuration of the
filter/cilium
processor that is built in to the Gloo telemetry pipeline. This configuration includes a regex that scrapes all metrics that start withhubble_
orcilium_
.filter/cilium: metrics: include: match_type: regexp metric_names: - hubble_.* - cilium_.*
Get your current values for the workload clusters.
helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $REMOTE_CONTEXT > data-plane.yaml open data-plane.yaml
In your Helm values file, enable the
metrics/cilium
pipeline that uses the processor that you reviewed in the previous step. Be sure thatglooNetwork.enabled
is also included in your values.glooNetwork: enabled: true telemetryCollector: enabled: true telemetryCollectorCustomization: pipelines: metrics/cilium: enabled: true
Upgrade your workload cluster installation by using your updated values file.
helm upgrade gloo-platform gloo-platform/gloo-platform \ --kube-context $REMOTE_CONTEXT \ --namespace gloo-mesh \ -f data-plane.yaml \ --version $GLOO_VERSION
Verify that your settings were added to the Gloo telemetry collector configmap.
kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context $REMOTE_CONTEXT
Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pods.
kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context $REMOTE_CONTEXT
Optional: To monitor your Cilium CNI, import the Cilium dashboard in Grafana.
Customize the Cilium metrics collection
Instead of enabling all Hubble and Cilium metrics that the Cilium agent emits, you can customize the Cilium processor and include only the metrics that you want to collect. All metrics are exposed on the Gloo telemetry collector agent where they can be scraped by the built-in Prometheus. You can view these metrics by using the Prometheus expression browser.
Single cluster
Get the Cilium and Hubble metrics that you want to collect. For an overview of supported metrics, see the Cilium documentation.
Get your current installation Helm values, and save them in a file.
helm get values gloo-platform -n gloo-mesh -o yaml > gloo-single.yaml open gloo-single.yaml
In your Helm values file, disable the
metrics/cilium
pipeline, and create a custom pipeline for the metrics that you want to collect. In the following example, thecilium_node_connectivity_status
andcilium_node_connectivity_latency_seconds
metrics are collected. Then, add your custom processor to themetrics/cilium
pipeline.telemetryCollector: enabled: true telemetryCollectorCustomization: extraProcessors: filter/cilium-custom: metrics: include: match_type: strict metric_names: # cilium hubble metrics to include - cilium_node_connectivity_status - cilium_node_connectivity_latency_seconds extraPipelines: metrics/cilium-custom: receivers: - prometheus processors: - memory_limiter - filter/cilium-custom - batch exporters: - otlp
Upgrade your installation by using your updated values file.
helm upgrade gloo-platform gloo-platform/gloo-platform \ --namespace gloo-mesh \ --version $GLOO_VERSION \ --values gloo-single.yaml
Verify that your custom Cilium settings were added to the Gloo telemetry collector configmap.
kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml
Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pod.
kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent
Optional: To monitor your Cilium CNI, import the Cilium dashboard in Grafana.
Multicluster
Get the Cilium and Hubble metrics that you want to collect. For an overview of supported metrics, see the Cilium documentation.
Get the Helm values files for your workload cluster.
helm get values gloo-platform -n gloo-mesh -o yaml --kube-context $REMOTE_CONTEXT > data-plane.yaml open data-plane.yaml
In your Helm values file, disable the
metrics/cilium
pipeline, and create a custom pipeline for the metrics that you want to collect. In the following example, thecilium_node_connectivity_status
andcilium_node_connectivity_latency_seconds
metrics are collected. Then, add your custom processor to themetrics/cilium
pipeline.telemetryCollector: enabled: true telemetryCollectorCustomization: extraProcessors: filter/cilium-custom: metrics: include: match_type: strict metric_names: # cilium hubble metrics to include - cilium_node_connectivity_status - cilium_node_connectivity_latency_seconds extraPipelines: metrics/cilium-custom: receivers: - prometheus processors: - memory_limiter - filter/cilium-custom - batch exporters: - otlp
Upgrade the workload cluster.
helm upgrade gloo-platform gloo-platform/gloo-platform \ --kube-context $REMOTE_CONTEXT \ --namespace gloo-mesh \ -f data-plane.yaml \ --version $GLOO_VERSION
Verify that your settings are applied in the workload cluster.
Verify that your settings were added to the Gloo telemetry collector configmap.
kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml --context $REMOTE_CONTEXT
Perform a rollout restart of the telemetry collector daemon set to force your configmap changes to be applied to the telemetry collector agent pods.
kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent --context $REMOTE_CONTEXT
Optional: To monitor your Cilium CNI, import the Cilium dashboard in Grafana.