Fluent Bit Plugin Troubleshooting Framework Log Forwarding

This guide is intended for help troubleshooting logs from the configuration and data collection end. For example: missing logs and/or log messages, unparsed logs, config questions etc. If the issue resides in the UI then there is a separate troubleshooting guide.

Before reading further:

  • Ensure that you have installed and configured a compatible Log Forwarder. You can find a list of the different log forwarding options and how to install New Relic’s various plugins in our documentation here. Troubleshooting for individual forwarders can be found below.
  • Logs in Context is different from Log Forwarding. Setting up Logs in Context for your application simply adjusts your application’s logger to format logs as JSON with New Relic’s logging metadata. This metadata establishes context for your application logs and other features of your New Relic APM agent. Check out our Configure logs in context with APM agents doc for more detailed information.

These are common information gathering and troubleshooting tips for the New Relic Fluentbit Plugin and not Fluentbit itself. If you are having trouble with Fluentbit then you will need to contact them for more information on their github: GitHub - fluent/fluent-bit: Fast and Lightweight Log processor and forwarder for Linux, BSD and OSX.

General

  • Verify that you have installed and setup up Fluentbit correctly using their documentation: Fluent Bit v1.7 Documentation - Fluent Bit: Official Manual
  • After Fluentbit is installed, ensure you have installed the New Relic Fluentbit Plugin using one of the methods mentioned in the Install the Fluentbit plugin section of our docs.
  • There are useful config examples in our public repo. These are also helpful for questions around adding custom attributes/fields and parsing/filtering.
  • Once the Plugin is installed, you will need to modify your configuration to have Fluentbit send collected logs to the Plugin by adding the following to your fluentbit.conf or td-agent-bit.conf configuration file:
[OUTPUT]

Name newrelic

Match *

licenseKey YOUR_LICENSE_KEY
  • Pay attention to white space when editing your config files. Be sure to use four spaces to indent and one space between keys and values. See Fluentbit’s documentation on Format and Schema for more information.
  • The Match * line is what determines the inputs to forward to the New Relic plugin. Using a wildcard matches all previously set Tags but this can also be refined to only specific inputs, see here for more information.

Troubleshooting

  • Check for errors using the following NRQL query:
    • For example: SELECT * FROM NrIntegrationError SINCE 24 hours AGO
  • If everything looks to be installed and configured correctly and you are still having issues then the next step is to generate some debug logs for Fluentbit and the output of your logs generated by Fluentbit. You can do so by modifying your fluentbit.conf or td-agent-bit.conf file and adding the following changes and config options. Make sure to restart the Fluentbit service for the changes to be applied:
[SERVICE]

Log_Level debug

Log_File /tmp/fluentbit_debug.log

[OUTPUT]

Name file

File fluentbit_output.log

Match *

Path /tmp
  • If you already have a [SERVICE] section, then add/adjust the settings shown above. This section is adjusting Fluentbit’s logging level to debug and creating a log file in your tmp directory.
  • The [OUTPUT] shown above is an additional output to be added to the config file below the New Relic Plugin’s output section. This output is matching all of your previous input sections and piping them to an output log that shows what Fluentbit is sending to New Relic.
  • You can look for errors or irregularities with Fluentbit itself in the fluentbit_debug.log file.
  • You can see exactly what Fluentbit is outputting to New Relic in the fluentbit_output.log file.
    • This is especially helpful if you are missing log lines or are confused by how your logs are being displayed in the New Relic. This file shows exactly what the fluentbit plugin is sending to New Relic.