Hello!

Recently, I worked on a task that involved receiving StatsD metrics and sending them to Prometheus. Since I was already using the Otel collector, I decided to utilize it for this task.

Otelcontrib provides a StatsD receiver (you can find it here: StatsD Receiver). Configuring the receiver itself is a relatively easy task. However, sending the received metrics to Prometheus proved to be more challenging. The reason for this is that the Prometheus Remote Write exporter drops StatsD metrics. Fortunately, I managed to solve this issue and would like to share my findings.

I attempted to set up Otel with StatsD in Kubernetes using the official Helm chart, as well as on an EC2 instance using RPM. The configuration itself remained the same in both cases. Below, I will share my example of setting it up on an EC2 instance.

To make it work, you first need a functioning Otel installation. It’s important to note that the StatsD receiver does not support scaling, so you can only have one instance of Otel with StatsD to receive data from a particular service. That’s why I added a separate instance of Otel with only the StatsD receiver, away from my other instances of Otel that have scaling enabled.

The configuration for the StatsD receiver is straightforward:

receivers:
  statsd:
    endpoint: 0.0.0.0:8125
    is_monotonic_counter: true

Next, I wanted to add an exporter such as the Prometheus Remote Write exporter:

  prometheusremotewrite:
    auth:
      authenticator: sigv4auth
    endpoint: https://PROMETHEUS/api/v1/remote_write
    resource_to_telemetry_conversion:
      enabled: true
    timeout: 30s

Here is my pipeline configuration:

    metrics:
      exporters:
        - logging
        - prometheusremotewrite
      processors:
        - memory_limiter
        - batch
        - cumulativetodelta
      receivers:
        - otlp
        - statsd

However, when I checked for data in Prometheus, I found nothing. This is because the Prometheus Remote Write exporter drops all Otel data. As mentioned in the Otel documentation:

⚠️ Non-cumulative monotonic, histogram, and summary OTLP metrics are dropped by this exporter.

I needed a way to make the metrics compatible, and another Otel exporter helped me achieve this. I introduced the Prometheus Exporter to my pipeline. This exporter exports data in the Prometheus format, allowing it to be scraped by a Prometheus server.

Here is the configuration for the Prometheus exporter:

  prometheus:
    enable_open_metrics: true
    endpoint: 0.0.0.0:9999
    metric_expiration: 3m
    namespace: statsd
    resource_to_telemetry_conversion:
      enabled: true
    send_timestamps: true

Now that I had the Prometheus exporter, I needed to write data to it by modifying the pipeline. In reality, I needed two pipelines, so I created the first one named “statsd”:

    metrics/statsd:
      exporters:
        - logging
        - prometheus
      processors:
        - memory_limiter
        - batch
        - cumulativetodelta
      receivers:
        - statsd

This pipeline receives data from StatsD and exports it as Prometheus metrics. I can now access my metrics using “MY_IP:9999/metrics”. My metrics are now in the Prometheus format. The next step is to scrape them and send them to the Prometheus backend. This can be achieved using the Prometheus receiver and scrape configuration:

  prometheus:
    config:
      scrape_configs:
        - job_name: statsd
          metric_relabel_configs:
            - replacement: dev
              target_label: env
          scrape_interval: 60s
          static_configs:
            - targets:
                - localhost:9999

Finally, I can build my pipelines, which will perform the following steps: data -> StatsD -> Prometheus endpoint -> scrape job -> Prometheus backend.

Here is the pipeline configuration in code:

    metrics/prometheus:
      exporters:
        - logging
        - prometheusremotewrite
      processors:
        - memory_limiter
        - batch
      receivers:
        - otlp
        - prometheus
    metrics/statsd:
      exporters:
        - logging
        - prometheus
      processors:
        - memory_limiter
        - batch
        - cumulativetodelta
      receivers:
        - statsd

The cumulativetodelta processor is required in the StatsD pipeline because it converts monotonic, cumulative sum, and histogram metrics to monotonic delta metrics. Non-monotonic sums and exponential histograms are excluded.

Processors also need to be defined in the configuration:

processors:
  batch: {}
  cumulativetodelta: null
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25

The final configuration will look like this:

exporters:
  logging: {}
  prometheusremotewrite:
    auth:
      authenticator: sigv4auth
    endpoint: https://PROMETHEUS/api/v1/remote_write
    resource_to_telemetry_conversion:
      enabled: true
    timeout: 30s
  prometheus:
    enable_open_metrics: true
    endpoint: 0.0.0.0:9999
    metric_expiration: 3m
    namespace: statsd
    resource_to_telemetry_conversion:
      enabled: true
    send_timestamps: true

extensions:
  health_check: {}
processors:
  batch: {}
  cumulativetodelta: null
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: opentelemetry-collector
          metric_relabel_configs:
            - replacement: statsd
              target_label: type
          scrape_interval: 10s
          static_configs:
            - targets:
                - localhost:8888
        - job_name: statsd
          metric_relabel_configs:
            - replacement: dev
              target_label: env
          scrape_interval: 60s
          static_configs:
            - targets:
                - localhost:9999
  statsd:
    endpoint: 0.0.0.0:8125
    is_monotonic_counter: true
service:
  extensions:
    - health_check
  pipelines:
    metrics/prometheus:
      exporters:
        - logging
        - prometheusremotewrite
      processors:
        - memory_limiter
        - batch
      receivers:
        - otlp
        - prometheus
    metrics/statsd:
      exporters:
        - logging
        - prometheus
      processors:
        - memory_limiter
        - batch
        - cumulativetodelta
      receivers:
        - statsd

As you can see in this configuration, there is an additional scrape job on port 8888. This is not required for StatsD, but this endpoint exposes Otel metrics. If you want to have the ability to monitor the status of your Otel installation, I suggest adding this scrape job.