Relic Solution: Infrastructure Log Forwarding - "Too Many Open File" Issue

Problem

We have seen cases were customers would enable the Infrastructure Agent Log Forwarder, but no logs would be forwarded to New Relic. In some cases customers would notice host Memory and/or CPU usage spike. Upon further investigation we noticed the Fluent Bit process was the culprit. In addition we noticed the following log entries:

level=error msg="[<path to some directory> errno=24] Too many open files" component=integrations.Supervisor output=stderr process=log-forwarder

Cause

This is an issue/limitation with the Fluent-Bit log forwarder, which is bundled with the Infrastructure Agent. Per the issues filed on the Fluent-Bit Github repo, there is a limit to how many logs Fluent-Bit can handle and doesn’t filter out stale logs:

The issue is not a New Relic issue, but a Fluent-Bit issue. We (New Relic) can’t speak to if/when the issue will be addressed by Fluent-Bit.

Solution

That being said, there are a few options:

  • scale down the number of log/directories in the file path. Fluent-Bit will attempt to tail all logs in the file path regardless if the logs are current/active (being written to). If there are a large number of logs, Fluent-Bit will run into the Too many open files error.

AND/OR

1 Like