Prometheus data loss while extends more 60 minutes

Hello!

I’ve set up an open Prometheus metrics integration to monitor my nomad cluster. When I try to view any metrics on a dashboard that exceed the 60 minute time window, they stop showing up. If I filter for 60 minutes or less it’s fine. Reference documentation used: https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-openmetrics/configure-prometheus-openmetrics-integrations/

Here is the examples and information:

The query for the dashboard is SELECT average(nomad_nomad_job_status_dead) FROM Metric FACET targetName SINCE 180 minutes AGO TIMESERIES :

Filtered 60m:

Filtered more than 60m , the data disappear and the reference change to 'metric’

Is there any configuration in the integration to manipulate these time windows?

Thanks a lot!

1 Like

Hi @lmontesoro

Thanks for reaching out, I hope you are well.

Congrats on your first post in the community, Whoop!

I have gone ahead looped in the Dashboards team for their insights as this is out of my scope, please note they will reach out via this post soon.

Also I did locate a New Relic resource How to monitor with Prometheus.

Should you have any additional questions or updates please do reach out.

1 Like

Hi @lmontesoro - Can you provide links to the queries in your screenshot? Thanks!

1 Like

Hi @dkoyano ! I quote the query!

UPDATE: This is the direct chart URL https://chart-embed.service.newrelic.com/herald/6bc48b96-48fe-47cd-b7fb-d32f09bcc6ee?height=400px&timepicker=true

And here, the dashboard short url: https://onenr.io/01wZq9GBAQ6

Thanks

@lmontesoro - It looks like you have some metric limits being hit on your account as shown in the query here. You will need to investigate and resolve these limits as documented below.

https://docs.newrelic.com/docs/data-apis/ingest-apis/metric-api/troubleshoot-nrintegrationerror-events/

2 Likes