Fluentd Plugin Troubleshooting Framework Log Forwarding

This guide is intended for help troubleshooting logs from the configuration and data collection end. For example: missing logs and/or log messages, unparsed logs, config questions etc. If the issue resides in the UI then there is a separate troubleshooting guide.

Before reading further:

  • Ensure that you have installed and configured a compatible Log Forwarder. You can find a list of the different log forwarding options and how to install New Relic’s various plugins in our documentation here. Troubleshooting for individual forwarders can be found below.
  • Logs in Context is different from Log Forwarding. Setting up Logs in Context for your application simply adjusts your application’s logger to format logs as JSON with New Relic’s logging metadata. This metadata establishes context for your application logs and other features of your New Relic APM agent. Check out our Configure logs in context with APM agents doc for more detailed information.

These are common information gathering and troubleshooting tips for the New Relic Fluentd Plugin and not Fluentd itself. If you are having trouble with Fluentd then you will need to contact them for more information on their github: GitHub - fluent/fluentd: Fluentd: Unified Logging Layer (project under CNCF).

General

  • Verify that you have installed and setup Fluentd correctly using their documentation: https://docs.fluentd.org/
  • After Fluentd is installed, ensure you have installed the New Relic Fluentd Plugin using one of the methods mentioned in the Install the Fluentd plugin section of our docs.
  • There are useful config examples in our public repo. These are also helpful for questions around adding custom attributes/fields and parsing/filtering.
  • Once the Plugin is installed, you will need to modify your configuration to have FluentD send collected logs to the Plugin by adding the following to your td-agent.conf or fluent.conf configuration file:
<match **>

@type newrelic

license_key xxxxxxxxxx

</match>
  • The “match **” line is what determines the inputs to forward to the New Relic plugin. ** matches everything but this can also be refined to only specific inputs, see here for more information.
    • There’s one tricky thing to note about these “match” configurations: Once a log line is correctly matched, it will not match any other directives. This means that you had some other plugin configured above ours with a <match **> (match everything) then the NR Plugin will never get data because it’s all being consumed by the earlier matches. This can be solved by using an “out_copy” directive. More information can be found in fluentd’s documentation on match order.
  • A common mistake is to mix up the api_key and license_key parameters in the tag for the plugin. Remember that the value for the api_key param is an Insights insert key and the value for the license_key param is your account license key.

Troubleshooting

  • Check for errors using the following NRQL query:
    • For example: SELECT * FROM NrIntegrationError SINCE 24 hours AGO
  • If everything looks to be installed and configured correctly and you are still having issues then the next step is to generate some debug logs for fluentd and the output buffer of your logs. You can do so by modifying your td-agent.conf or fluent.conf file and adding the following changes and config options:
<system>

log_level debug

</system>

<match fluent.**>

@type file

path /tmp/fluentd_debug_buffer

</match>

<match **>

@type copy

<store>

@type file

path /tmp/fluentd_output_buffer

</store>

<store>

@type newrelic

license_key xxxxxxxxxxxxxx

</store>

</match>
  • sets the log level for fluentd

  • <match fluent.**> matches those fluentd debug logs and sends them as a buffer file into the /tmp/fluentd_debug_buffer directory. By default, these logs also show in stdout for fluentd but it can be handy to have a file to go over as well.

  • @type copy and make it so you can write to multiple outputs. The first tag is sending an output buffer file of your logs to /tmp/fluentd_output_buffer. The second is sending them to New Relic.

  • You can look for errors or irregularities with fluentd itself in the buffer.xxxx.log file in the /tmp/fluentd_debug_buffer directory.

  • You can see exactly what Fluentd is outputting to New Relic in the buffer.xxxx.log file in the /tmp/fluentd_output_buffer directory.

    • This is especially helpful if you are missing log lines or are confused by how your logs are being displayed in the New Relic. This file shows exactly what the fluentd plugin is sending to New Relic.