Hi, part of my day job is protecting our customers’ applications from abuse by scrapers. One of our customers is using Cloudflare for it, but this topic is not Cloudflare-specific per se.
New Relic Synthetics is using the user agent “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3282.275 Safari/537.36”, but is not actually using a browser. Because it is a ping check it acts just like an unauthorized scraper would, and because it’s not using a browser it’s failing Cloudflare’s JS challenge.
If you were to announce yourselves using a custom user agent such as “New Relic Synthetics”, you would be easily recognizable.
In a previous topic, your reply to a similar question has been that there are IP addresses you publish that we are supposed to be whitelisting, and that setting a custom header might be an option, however there are a number of issues with that:
- You are exhibiting some of the same behavior malicious scrapers are by hiding who you are.
- It is not possible to allow requests in Cloudflare based on custom headers.
- It doesn’t seem possible to even set custom headers anymore in the interface.
- I could write a custom script, but why should I have to write a script for what is a simple ping check? Surely a simple ping check should be manageable for non-programmers.
- It’s a lot less effort for you to set a custom user agent, than the combined effort for all of your customers to maintain IP whitelists.
Could you please set a sensible default user agent, or allow us to easily change it in simple ping monitors?
New Relic Edit
- I want this too
- I have more info to share (reply below)
- I have a solution for this
We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.