Prometheus "instant" query is not stable

We are evaluating your new feature of prometheus managed server.
we are sending prometheus metrics to new relic via the remote_write integration (add more data->prometheus) and we are seeing some issues when integrating with Grafana.

To be more specific, there is a significant instability when running “instant” query in Grafana Labs using NR prometheus as data source, range queries works ok.
but this is not just in Grafana, querying the metrics using the “instant” checkbox in New Relic “query your data” section with promql-style query yield the same non-stable result,
sometimes the data is returned and sometimes not (zero results).
This happens with every query, even the simplest like node_load5

we really want to go all the way and start shipping all of our customers prometheus metric (Tera bytes scale a month) to New Relic managed prometheus but right now this issues is a show stopper

Hi Shmulik, can I ask what scrape interval you are using for your prometheus installation?

1 Like

Looks like New Relic translates promql queries to NRQL, specifically promql queries of the type “instant”
are translated to “SINCE 1 MINUTES AGO UNTIL NOW” NRQL queries.
Since we are pushing the prometheus metrics every 2 minutes, a one minute gap is opened.

This translation to get only metrics from the last minute is not the expected prometheus behavior.
According to Grafana and prometheus docs , “instant” queries should return only the latest value prometheus has scraped for the requested time series without being bound to a time range (See screenshot).

Could this be fixed on the New Relic side to fit prometheus standards?