Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Generate PageView count by UserAgendString (to detect Bot Traffic)


#1

Hi guys -

I want to create a list of page view counts grouped by raw user agent string to detect bot traffic. I came up with this statement which returns an empty result. Obviously, my solution is wrong :slight_smile:

SELECT count(*) FROM PageView FACET request.headers.userAgent SINCE 1 day ago

Can you point me into the right direction?

Thanks!
Michael


#2

Hey @michael.romer! You’re definitely on the right track, but the issue you are hitting is that request.headers.userAgent isn’t one of the default attributes on PageView events - Insights will return empty values like you’re experiencing if this happens. There’s a couple options available depending on what works best for you:

  • Utilize the existing attributes on PageView - By default, PageView events automatically collect UserAgent Name, OS, and Version. If this is all you need to find out if traffic to your site is bot-related, then you can amend your Insights query: SELECT count(*) FROM PageView FACET userAgentName, userAgentVersion, userAgentOS SINCE 1 day ago
  • Add the raw string as a custom attribute - Using the setCustomAttribute() API call right when the page loads, you can capture the full user agent string and add this as a custom attribute to PageView events that are happening. You can then see that data in Insights and be able to narrow down the scope to see more about where traffic is coming from using the LIKE feature.

Let us know if this works for you or if you have any other questions!


Detect search bots crawling stats