Logstash Plugin Troubleshooting Framework Log Forwarding

This guide is intended for help troubleshooting logs from the configuration and data collection end. For example: missing logs and/or log messages, unparsed logs, config questions etc. If the issue resides in the UI then there is a separate troubleshooting guide.

Before reading further:

  • Ensure that you have installed and configured a compatible Log Forwarder. You can find a list of the different log forwarding options and how to install New Relic’s various plugins in our documentation here. Troubleshooting for individual forwarders can be found below.
  • Logs in Context is different from Log Forwarding. Setting up Logs in Context for your application simply adjusts your application’s logger to format logs as JSON with New Relic’s logging metadata. This metadata establishes context for your application logs and other features of your New Relic APM agent. Check out our Configure logs in context with APM agents doc for more detailed information.

These are common information gathering and troubleshooting tips for the New Relic Logstash Plugin and not Logstash itself. If you are having trouble with Logstash then you will need to contact Elastic for more information: Have questions? Contact Elastic | Elastic.

General

  • Verify that you have installed and setup Logstash correctly using their documentation: Installing Logstash | Logstash Reference [7.12] | Elastic
  • After Logstash is installed, ensure you have installed the New Relic Logstash Plugin using one of the methods mentioned in the Logstash plugin for log forwarding section of our docs.
  • There are useful config examples in our public repo. These are also helpful for questions around adding custom attributes/fields and parsing/filtering.
  • Once the NR Logstash plugin is installed it needs to be configured. The exact location where you will need to add this configuration depends on your environment because it is possible to define custom configuration files when starting logstash. In general though, the following needs to be included in your logstash.conf file:
output {

newrelic {

license_key => "LICENSE_KEY"

}

}
  • You can also use an Insert API Key defined as api_key in the config in place of your license_key.
  • The Optional Configuration section of our docs lists a couple of optional configurations that could also be used. The main one to note is base_uri which will be necessary if you are based in the EU.

Troubleshooting

  • Check for errors using the following NRQL query:

    • For example: SELECT * FROM NrIntegrationError SINCE 24 hours AGO
  • If everything looks to be installed and configured correctly and you are still having issues then the next step is to look over Logstash’s logs and an output file of your logs.

  • You can generate an output log file to double check how Logstash is processing your logs by adding a file output like so:

output {

newrelic {

license_key => "LICENSE_KEY"

}

file {

path => "/tmp/logstash_output.log"

codec => line

}

}
  • You can also check for errors in Logstash’s logs as well. They are located at /var/log/logstash/logstash-plain.log
    • If you need to see a more verbose level of logging for logstash you can use the --log.level=debug command flag when starting logstash.