Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Troubleshooting New Relic's Log Forwarder Plugins

new-relic-logs

#1

Introduction

New Relic’s Log forwarder plugins provide a great way to send relevant log information over to New Relic. However, whether you’re new to log forwarders (fluentd, logstash, etc) or an old-hand, it can be difficult to know where to start when investigating potential issues.

In this post I’ll try to present the best way to begin troubleshooting and, if that doesn’t help resolve your issue, the types of data we in New Relic Support would like to see when you reach out in order to quickly investigate. The steps in this document mostly pertain to New Relic Plugins for FluentD, Logstash, and Fluent-bit.

Troubleshooting Steps

1. Are there any errors recorded in New Relic?

This step is fairly straightforward. Head on over to Insights and run the following query:

SELECT * from NrIntegrationError where newRelicFeature = 'Logs' since 7 days ago

Any results you see will be directly related to an attempted log ingestion. The most common message you’ll see here is: “Error parsing JSON payload” - At the moment we do not store these failed payloads however step 3 below may help in identifying the issue. This error is most common when using the Log API.

2. Is New Relic responding with a 403 response code?

When you attempt to send logs to New Relic, our ingest endpoint will respond with a status code depending on the result. A 202 Response Code means your license key/API key was accepted. However, if you receive a 403 response (or other related error code) then the logs did not make it to our system.

Most of our plugins do not log the response if it is a 202. However, in most cases an error response code(403 or similar) will be shown in the logs for your particular log forwarder.

The most common cause of a 403 response is an incorrect license key or API key. It’s worth double-checking that you aren’t using your license key and labeling as an API key, or vice versa!

Please keep in mind that payload verification occurs after the logs are accepted into our system; So a 202 response does not guarantee that everything went well. It is just the first in a line of steps.

3. Is the log forwarding tool correctly configured?

The best way to determine whether your log forwarder has been correctly configured is to copy the output that goes to the New Relic Plugin, into a file. This allows you to verify any formatting is occurring as expected before that data reaches our plugin. It also allows you to ensure that your input is actually picking up log records(if not, the file will remain empty). Please save this file and attach it to your Support ticket if one becomes needed. The steps for doing this will vary by Log Forwarder:

FluentD

In FluentD, the order of your <match> and <filter> directives matters (see this FluentD documentation) so it is important to use a <copy> directive for this step. For example, the following configuration will send matched records to both New Relic and the configured file. For the most accurate results use the same <match TAG> as you were using with just the New Relic plugin configured:

<match **>
  @type copy
  <store>
    @type file
    path /path/to/output_test.log
  </store>
  <store>
    @type newrelic
    license_key YOUR_LICENSE_KEY
  </store>
</match>
Logstash

Add the below “file {…}” configuration to your existing output:

output {
  newrelic {
    license_key => "LICENSE_KEY"
  }
  file {
    path => /path/to/output_test.log
    codec => line
  }
}
Fluent-bit

Add the following configuration below the newrelic OUTPUT entry:

[OUTPUT]
    Name file
    Match *
    Path /path/to/output_test.log

Once that has been setup, try to reproduce the issue you were seeing. After viewing the results of the copied output it should become apparent if there are any issues with your local configuration. If the entries do not look as expected, or no entries show up at all, you’ll want to go back through your configuration to find the disconnect.

If everything looks good as far as you can tell, save this file in case you need to create a Support ticket.

Next steps

Still running into issues? No problem, we’re happy to help! Feel free to open a support ticket or make a post here in the Explorer’s Hub so that others can benefit from the solution once it’s found. The exact data necessary for troubleshooting will vary depending on your issue but in general we’d like to see one or two items for any Logs related tickets:

  • A copy of your configuration file
  • A copy of the records being sent to New Relic (see Troubleshooting step 3 above)
  • A copy of your log forwarder’s log file

More information may be necessary depending on the issue, but the above gives us everything we need to immediately start digging in.

Resources and Common Issues