Restarts and gaps in data (Kubernetes)

Problem

When running the Prometheus OpenMetrics integration for Kubernetes, you notice restarts and gaps in data sent to New Relic.

Solution

When running the Prometheus OpenMetrics integration for Kubernetes with 500K data points per minute, be sure to set these limits:

• CPU limit: 1 core
• Memory limit: 1Gb

Recommendation: Always run the Kubernetes scraper with one replica.

Adding more replicas will result in duplicated data. If the CPU and memory limits are not sufficient, this can result in restarts and gaps in the data.

To check the status and restart events for the scraper:

kubectl describe pod -l "app=nri-prometheus"