Feature Idea: New Relic acts like a scraper because it uses a browser UA

Hi, part of my day job is protecting our customers’ applications from abuse by scrapers. One of our customers is using Cloudflare for it, but this topic is not Cloudflare-specific per se.

New Relic Synthetics is using the user agent “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3282.275 Safari/537.36”, but is not actually using a browser. Because it is a ping check it acts just like an unauthorized scraper would, and because it’s not using a browser it’s failing Cloudflare’s JS challenge.

If you were to announce yourselves using a custom user agent such as “New Relic Synthetics”, you would be easily recognizable.

In a previous topic, your reply to a similar question has been that there are IP addresses you publish that we are supposed to be whitelisting, and that setting a custom header might be an option, however there are a number of issues with that:

  1. You are exhibiting some of the same behavior malicious scrapers are by hiding who you are.
  2. It is not possible to allow requests in Cloudflare based on custom headers.
  3. It doesn’t seem possible to even set custom headers anymore in the interface.
  4. I could write a custom script, but why should I have to write a script for what is a simple ping check? Surely a simple ping check should be manageable for non-programmers.
  5. It’s a lot less effort for you to set a custom user agent, than the combined effort for all of your customers to maintain IP whitelists.

Could you please set a sensible default user agent, or allow us to easily change it in simple ping monitors?

Thanks,

Toon Spin


New Relic Edit

  • I want this too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.

Hi @toon.spin -

Thanks for posting with so much detail! Right now ping monitor user agents can not be edited by NR users. So I’ll file a feature request internally for you, for that. Adding in all of your points makes for a great argument for editable user agents, so hopefully this will all be taken on board by the Synthetics Product manager.

To add in some additional detail, Synthetics sends it’s identity in an X-ABUSE-INFO header, it does that for all monitor types. I see in your 2nd point that this doesn’t work for you with Cloudflare, but I did want to make sure folks viewing this post later know about it, in case this can work for them.

We’ll update you here if we get an update on this feature request!

3 Likes

Hi Ryan,

Thanks for picking this up.

I was not aware of the X-Abuse-Info header - as you remarked that’s not particularly relevant to my own situation but I agree that it might help somebody else out - always good to help out any stragglers stumbling upon this sort of thread from Google!

Also it might actually help me add better Synthetic ping monitors to other sites we have behind our own homebrewed IP whitelists.

Toon

2 Likes

Glad that info was helpful - full details here:

https://docs.newrelic.com/docs/synthetics/new-relic-synthetics/administration/identify-synthetics-requests-your-app

2 Likes