Infrastructure Troubleshooting Framework Log Forwarding

This guide is intended for help troubleshooting logs from the configuration and data collection end. For example: missing logs and/or log messages, unparsed logs, config questions etc. If the issue resides in the UI then there is a separate troubleshooting guide.

Before reading further:

  • Ensure that you have installed and configured a compatible Log Forwarder. You can find a list of the different log forwarding options and how to install New Relic’s various plugins in our documentation here. Troubleshooting for individual forwarders can be found below.
  • Logs in Context is different from Log Forwarding. Setting up Logs in Context for your application simply adjusts your application’s logger to format logs as JSON with New Relic’s logging metadata. This metadata establishes context for your application logs and other features of your New Relic APM agent. Check out our Configure logs in context with APM agents doc for more detailed information.

General

  • Ensure that you are using Infrastructure agent version 1.11.4 or higher.
  • Log forwarding with Infrastructure only works with the operating systems listed in the Infrastructure logging docs under What you need. Keep in mind that log forwarding does not work when the Infrastructure agent is containerized.
  • Make sure you have configured a yml file in your logging.d directory:
    • Linux: /etc/newrelic-infra/logging.d/
    • Windows: C:\Program Files\New Relic\newrelic-infra\logging.d\
      1. The Agent will try to parse configuration data from any file ending in .yml or .yaml found in your logging.d directory.
      2. The Agent will automatically detect changes to these files and update the related fluentbit configuration without requiring a restart of the Agent itself EXCEPT if you are using an extra fluent bit config or parser as defined under the fluentbit parameter. The agent will not automatically register changes made to those files and needs to be restarted manually.
      3. Double check that the formatting of your yml file is valid and has proper spacing. Use a yml validator if you are unsure of the format.
      4. Ensure you are using valid parameters for the config by checking against our Log forwarding parameters for Infrastructure logging.
      5. Windows: The log file path in the yml needs to have double backslashes and quotes around the path.

Troubleshooting

  • Check for errors using the following NRQL query:

    • For example: SELECT * FROM NrIntegrationError SINCE 24 hours AGO
  • Set the Infrastructure logs to verbose and look for errors. You can adjust the log level for Infrastructure by using the instructions in our Generate logs for troubleshooting the infrastructure agent documentation.

    • Once you have the verbose logs you will need to search the file for lines containing: integrations.Supervisor. These log lines are related to log forwarding.
  • The Infrastructure agent uses fluentbit to forward logs to New Relic. When troubleshooting it can be useful to look at debug logs for the fluentbit process as well as the output buffer of the logs that are processed by fluentbit. To generate these, you will need to add some additional config files to your logging.d directory.

Linux

Create a fluentbit.conf file and add the following:

[SERVICE]

Log_Level debug

Log_File /tmp/fluentbit-debug.log

[OUTPUT]

Name file

File fluentbit-output.txt

Match *

Path /tmp

* Create a new yml file called fluentbit.yml and add the following:

logs:

- name: fluentbit-import

fluentbit:

config_file: /etc/newrelic-infra/logging.d/fluentbit.conf

Windows:

Create a fluentbit.conf file and add the following:

[SERVICE]

Log_Level debug

Log_File C:\temp\fluentbit-debug.log

[OUTPUT]

Name file

File fluentbit-output.txt

Match *

Path C:\temp

* Create a new yml file called fluentbit.yml and add the following:

logs:

- name: fluentbit-import

fluentbit:

config_file: /etc/newrelic-infra/logging.d/fluentbit.conf

If you’re using Windows and you are missing a specific redistributable you will see the runtime errors described in our Runtime error on Windows docs. You need to install Visual Studio 2015, 2017 and 2019 to fix this. Please note that this is an all-in-one redistributable and is different from each of the individual versions.

1 Like