Forward data to Datadog

Set up an extra exporter to forward pipeline metrics to your Datadog instance.

If you want to set up an exporter for a different destination, update your configuration in the values file for the installation Helm chart accordingly, and use the same steps as outlined in this topic to update your pipeline. You can find an overview of supported providers in the OpenTelemetry documentation.

  1. Get the URL and the API key to log into your Datadog instance. For more information about setting up an API key, see the Datadog documentation.

  2. Add the datadog site and API key information to your Gloo Platform Helm values file. When you add this configuration, the default metrics are forwarded to your Datadog instance. You can configure your Gloo OTel pipeline to collect more metrics, such as for the Gloo management server and Cilium CNI and make these metrics available to the telemetry gateway so that they can be forwarded to Datadog.

            site: <datadog-site>  # Example:
            key: <datadog-api-key>
          - otlp # Metrics received by the Collector in workload clusters.
          - memory_limiter
          - batch
          - datadog # Exporter specified above as the new destination.
  3. Follow the Upgrade Gloo Mesh guide to apply the changes in your environment.

    Multicluster environments only: Choose whether to forward telemetry data from the OTel gateway or from the OTel collectors.

    • OTel gateway on management cluster: Because the Gloo telemetry gateway receives metrics from all workload clusters, you might decide to forward the metrics from the telemetry gateway to Datadog. Use the Helm values file for the management cluster to add the exporter configuration to the Gloo telemetry gateway. Then, apply the changes by upgrading the management cluster. No upgrade of any workload cluster is required.
    • OTel collectors on workload clusters: If you have several workload clusters, forwarding telemetry data from the telemetry collectors instead of the telemetry gateway might be a more reliable and robust method for data forwarding. To apply the changes to a Gloo telemetry collector agent in one or all workload clusters, you must update the Gloo agent with the settings in your values file for the Helm release in the workload cluster. Note that you need to replace telemetryGatewayCustomization with telemetryCollectorCustomization in your Helm values file so that your changes can be applied to the OTel collector agents in your workload clusters.
  4. Depending on where you applied the changes, verify that the configmap for the telemetry gateway or collector agent pods is updated with the values you that set in the values file.

    kubectl get configmap gloo-telemetry-gateway-config -n gloo-mesh -o yaml
    kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh -o yaml
  5. Depending on where you applied the changes, perform a rollout restart of the gateway deployment or the collector daemon set to force your configmap changes to be applied to the telemetry gateway or collector agent pods.

    kubectl rollout restart -n gloo-mesh deployment/gloo-telemetry-gateway
    kubectl rollout restart -n gloo-mesh daemonset/gloo-telemetry-collector-agent