Auto Detecting Robot Vs Crawlers by scanning and flagging IPs/ set of IPs requesting APM.
We observed a spike quite often due to scrapping/spamming and I assume the same must be happening to others as well. With some DNS API if it shows IPs globally attacking and affecting many of us will help us nail down them not just by blocking IPs but introducing some methods to filter them out with message 429 "Too Many Requests"
Actually here NewRelic would help us by collecting and showing the information like
- Suspicious Spammer
- NA - Normal User/ Group of users
- XYZ - Company Load Test Tool
Different companies logging the data in NewRelic can also help by tagging some IPs.
I am assuming that this will reduce many unwanted attacks.
We tried to add a custom attribute to see IP as suspicious spammer
- NRQL queries I have tried so far:
SELECT count(*) FROM Transactions FACET appName, httpHost , remoteIpAddress SINCE 1 day ago TIMESERIES
We tried to add a custom attribute to see IP as suspicious spammer but may not help other companies whereas they might be also have been getting spam from the same IP(s).
So, it would be similar to Google Ad Sense Service alert and not allow the owner to proceed further until their domain / IP is whitelisted