Forward data to Datadog

Set up an extra exporter to forward pipeline metrics to your Datadog instance.

1. Get the URL and the API key to log into your Datadog instance. For more information about setting up an API key, see the Datadog documentation.

2. Add the datadog site and API key information to your Gloo Platform Helm values file. When you add this configuration, the default metrics are forwarded to your Datadog instance. You can configure your Gloo OTel pipeline to collect more metrics, such as for the Gloo management server and Cilium CNI and make these metrics available to the telemetry gateway so that they can be forwarded to Datadog.

   telemetryGatewayCustomization:
     extraExporters:
       datadog:
         api:
           site: <datadog-site>  # Example: datadoghq.eu
           key: <datadog-api-key>
     extraPipelines:
       metrics/workload-clusters:
         receivers:
         - otlp # Metrics received by the Collector in workload clusters.
         processors:
         - memory_limiter
         - batch
         exporters:
         - datadog # Exporter specified above as the new destination.
   

3. Follow the Upgrade Gloo Mesh guide to apply the changes in your environment.
   
   Multicluster environments only: Because the Gloo telemetry gateway has all telemetry data for your workloads, Gloo Platform, and metrics pipeline, new exporter configurations are ideally added to the Gloo telemetry gateway in the management cluster so that you can forward all telemetry data to your preferred destination. To apply the changes, you must upgrade only the Gloo management server with the updated values file. No upgrade of the Gloo agent is required. To optionally apply the changes to a Gloo OTel collector agent in one or all workload clusters, you must update the Gloo agent with the settings in your values file for the Helm release in the workload cluster. Note that you need to replace telemetryGatewayCustomization with telemetryCollectorCustomization in your Helm values file so that your changes can be applied to the OTel collector agents in your workload clusters.

4. Depending on where you applied the changes, verify that the configmap for the telemetry gateway or collector agent pods is updated with the values you that set in the values file.

   kubectl get configmap gloo-telemetry-gateway-config -n gloo-mesh --context $MGMT_CONTEXT -o yaml
   

   kubectl get configmap gloo-telemetry-collector-config -n gloo-mesh --context $REMOTE_CONTEXT1 -o yaml
   

5. Depending on where you applied the changes, perform a rollout restart of the gateway deployment or the collector daemon set to force your configmap changes to be applied to the telemetry gateway or collector agent pods.

   kubectl rollout restart -n gloo-mesh --context $MGMT_CONTEXT deployment/gloo-telemetry-gateway
   

   kubectl rollout restart -n gloo-mesh --context $REMOTE_CONTEXT1 daemonset/gloo-telemetry-collector-agent