View and query your Prometheus data

To query and visualize the metrics collected for your Prometheus OpenMetrics or remote write integration with New Relic, you can use NRQL. You can also translate your PromQL-style queries to NRQL using either Grafana or the query builder.

All metrics for Docker and Kubernetes are stored in the Metric type.

Default attributes for the OpenMetrics integration

By default, the following attributes will be added to all metrics for Docker and Kubernetes integrations:

Kubernetes: If the scraper is running in Kubernetes, New Relic also adds the following attributes to all the metrics:

Default attributes for the remote write integration

By default, the following attributes will be added to Prometheus remote write metrics:

NRQL query examples

When you build queries, be aware that there is no linking between the metrics, entities, and attributes. Use the following NRQL queries to find out which metrics are available and which attributes are present on these metrics:

FROM Metric SELECT uniques(metricName)

FROM Metric SELECT uniques(metricName) WHERE instrumentation.provider='prometheus' AND instrumentation.name='remote-write'

FROM Metric SELECT uniques(metricName) WHERE instrumentation.provider='prometheus' AND instrumentation.name='remote-write' AND instrumentation.source='<ds>'

FROM Metric SELECT uniques(metricName) WHERE scrapedEndpoint='<ep>'

FROM Metric SELECT uniques(metricName) WHERE clusterName='<cn>'

FROM Metric SELECT uniques(metricName) WHERE namespaceName='<ns>'

FROM Metric SELECT uniques(metricName) WHERE podName='<pod>'

FROM Metric SELECT keyset() WHERE metricName='<mn>'

FROM Metric SELECT uniques(<attribute>) WHERE metricName='<mn>' AND podName='<pod>'

Build the query

Using metric name and attributes, you can query your data. For more information about facets, time series, and time selection, see the NRQL documentation.

To build PromQL-style queries, see our docs.

FROM Metric SELECT <metricName> WHERE <attribute>='<value>'

FROM Metric SELECT <aggregator>(<metricname>) WHERE <attribute>='<value>' TIMESERIES

FROM Metric SELECT latest(redis_connected_clients) WHERE clusterName='my-cluster' FACET scrapedEndpoint TIMESERIES

FROM Metric SELECT latest(redis_connected_clients) WHERE namespaceName='default' FACET podName TIMESERIES

FROM Metric SELECT average(node_memory_MemFree_bytes) WHERE clusterName='my-cluster'

FROM Metric SELECT average(container_memory_usage_bytes) WHERE deploymentName='my-app-deployment' AND namespaceName='default'

View data in New Relic

When you query the data, you can view the results in the New Relic UI. You can also visualize the data as charts, histograms, etc.

To view the NRQL query results for your Prometheus integration's data: Go to one.newrelic.com > Query your data. For more information, see New Relic's query builder documentation.

Create histograms and summaries

With remote write or version 1.2.0 or higher of the Prometheus OpenMetrics integration, you can create histograms and percentiles (summaries) of your data. The OpenMetrics data is based on New Relic's guidelines in GitHub for higher level metric abstractions, while the remote write data closely matches the schema of the original Prometheus data.

NRQL has two functions that work on remote write ingested PromQL: bucketPercentile() and histogram(). The links include query examples.These two functions don't work on OpenMetrics ingested buckets.

To better support visualization of histograms, percentiles are calculated based on the histogram metrics and sent to New Relic. To configure the calculated percentiles for OpenMetrics, use the percentiles configuration option.