Nri-prometheus scraping issue

Support: Full-Stack Observability (FSO) Infrastructure
#1

Hello all,

I have an issue with a Prometheus endpoint that gives some errors upon scraping.
I have annotated the pod for scraping and tested the scraping endpoint from inside the container and is working fine.
This is the warning that the nri-prometheus agent is throwing:

time=“2022-05-26T14:53:29Z” level=warning msg=“fetching Prometheus metrics: http://10.124.3.5:8024/actuator/prometheus (axonserver-0-0)” component=Fetcher error=“text format parsing error in line 189: invalid escape sequence '\”’"
time=“2022-05-26T14:53:29Z” level=warning msg=“error while scraping target” component=Fetcher error=“text format parsing error in line 189: invalid escape sequence '\”’"

Any idea what may have cause this?

Thanks!

#2

Switching to Prometheus with remote_write is working OK, but I would have preferred to use the nri-Prometheus deployment instead.

#3

@florin.chircea :wave:
I believe this error is coming from the official Prometheus parser which we use for parsing:
common/text_parse.go at main · prometheus/common · GitHub.
This seems to indicate that the character sequence is indeed not allowed.
The set of allowed characters for Prometheus metric and label names is spec’d here:

It must match the regex [a-zA-Z_:][a-zA-Z0-9_:]*

The escape \ characters are not allowed. I’m surprised to hear this would work with the remote-write as nri-prometheus uses the same parser Prometheus server uses. Perhaps the remote-write discards these metrics without giving a warning.

Anyhow, I would suggest reporting this to the author of the Prometheus exporter for the application in question so that it can be fixed at the right spot.

Hope this helps!

#4

I’m having a similar issue, but the invalid escape sequence '\\\"' is triggered by having quotes in #HELP lines, e.g.

# HELP system_cpu_usage The "recent cpu usage" of the system the application is running in
# TYPE system_cpu_usage gauge
system_cpu_usage 0.07835343215753132

I am deploying the nri-bundle helm chart version 3.4.0 in Kubernetes.

Is this also expected behaviour? If so, is there a workaround?

#5

Hi @aaron.verachtert1

Thanks for reaching out, I hope you are well.

I have gone ahead and looped in the Infrastructure support team here, as unfortunately this is out of my scope.

Please note they will reach out via this post with their findings.

#6

More info into my specific problem. The issue that is opened here was raised by my request.

#7

Thank you for the clarification @florin.chircea

Our support team is looking into this and will get a response out to you shortly. Please reach out if there are any changes on your end, or if you have any other questions as well.

#8

Our engineering team has been able to reproduce this issue and is currently working on a fix. As soon as they have a fix implemented and I will provide an update on this post. :grinning:

#9

A new version of the nri-prometheus image v2.16.2 has been published and is included in the new version of the chart and nri-bundle-4.7.1 as well.