No logging.d directory - what to try? (EC2 instance, not container)

We have a number of EC2 instances where I have log forwarding working correctly. Then there are some where it does not work. The New Relic infrastructure agent was installed by someone who is no longer here, and I don’t know what options they may have used when they did the install. Both the working and not-working systems have version 1.16.3.

On the systems that don’t work, /etc/newrelic-infra/ exists but /etc/newrelic-infra/logging.d/ does not. /var/db/newrelic-infra/newrelic-integrations/ exists but /var/db/newrelic-infra/newrelic-integrations/logging/ does not. There is no fluent-bit process running.

I tried creating /etc/newrelic-infra/logging.d/, setting the permissions to match a working system, and copying in my .yml file. It looks fine but the logs are never forwarded to New Relic.

These are EC2 instances, not containers.
$ cat /etc/os-release
NAME=“Amazon Linux AMI”
VERSION=“2018.03”
ID=“amzn”
ID_LIKE=“rhel fedora”
VERSION_ID=“2018.03”
PRETTY_NAME=“Amazon Linux AMI 2018.03”
ANSI_COLOR=“0;33”
CPE_NAME=“cpe:/o:amazon:linux:2018.03:ga”
HOME_URL=“http://aws.amazon.com/amazon-linux-ami/

I looked at the Fluent Bit docs at Fluent Bit plugin for log forwarding | New Relic Documentation but got stuck at step 1 because I don’t know how to find “your plugins directory”.

So my question is: what’s the best way forward? Try to install fluentbit, or reinstall the New Relic infrastructure agent, or ?

Thanks,
Steve

It might be worth trying to update the version of the agent. You are a few behind. Checking the update or install log for problems creating those folders is a likely next step.

At this point /var/log/newrelic-infra/ is empty and /etc/newrelic-infra.yml does not contain any logging settings (only the license key).

I created a ticket to have the infrastructure agent upgraded. It will likely be a couple of weeks before that gets done.

Hi @scorwin,

Thank you for the update. Once the infrastructure agent is updated on your end, let us know if you are still having issues and where we can jump in to help.

We installed the latest infrastructure agent last night, but that did not create a logging.d directory. The details are below. What else can we try?

Thanks,
Steve

$ sudo yum install newrelic-infra
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main | 2.1 kB 00:00:00
amzn-updates | 3.8 kB 00:00:00
Resolving Dependencies
→ Running transaction check
—> Package newrelic-infra.x86_64 0:1.16.3-1.el6 will be updated
—> Package newrelic-infra.x86_64 0:1.20.7-1.el6 will be an update
→ Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================
Package Arch Version Repository Size
===========================================================================================================================
Updating:
newrelic-infra x86_64 1.20.7-1.el6 newrelic-infra 35 M

Transaction Summary
===========================================================================================================================
Upgrade 1 Package

Total download size: 35 M
Is this ok [y/d/N]: y
Downloading packages:
newrelic-infra-1.20.7-1.el6.x86_64.rpm | 35 MB 00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : newrelic-infra-1.20.7-1.el6.x86_64 1/2newrelic-infra start/running, process 1975
newrelic-infra stop/waiting
Cleanup : newrelic-infra-1.16.3-1.el6.x86_64 2/2
chdir /var/db/newrelic-infra/
newrelic-infra start/running, process 27344
Verifying : newrelic-infra-1.20.7-1.el6.x86_64 1/2
Verifying : newrelic-infra-1.16.3-1.el6.x86_64 2/2

Updated:
newrelic-infra.x86_64 0:1.20.7-1.el6

Complete!

@scorwin
I would recommend fully uninstall the newrelic agent and also delete the newrelic directories.

  1. Stop the agent service:
  • sudo systemctl stop newrelic-infra.service
  1. Remove the data directory:
  • sudo rm -r /var/db/newrelic-infra/data
  • sudo rm -r /etc/newrelic-infra
  1. Use either apt (Debian or Ubuntu) or yum (Amazon Linux, CentOS, or RHEL) depending on the OS in question:
  • sudo {apt-get|yum} remove newrelic-infra

Then as documented in Install the infrastructure monitoring agent for Linux
Use the Guided Install which will generate an install command similar to the below ( or just copy and paste below with your License Key and Account ID):

curl -Ls https://download.newrelic.com/install/newrelic-cli/scripts/install.sh | bash && sudo NEW_RELIC_API_KEY=XXXXXXX NEW_RELIC_ACCOUNT_ID=XXXXXX /usr/local/bin/newrelic install

During the installation, please make sure that the below Log integration option is selected.

Please choose from the following instrumentation to be installed:  [Use arrows to move, space to select, <right> to all, <left> to none, type to filter]
> [x]  Logs integration
  [x]  Golden Signal Alerts

I just have tested it on Amazon Linux and the logging.d directory has been created.

ls -ltr
total 0
drwxr-xr-x 2 root root  31 Jan 12 14:57 integrations.d
drwxr-xr-x 2 root root 176 Jan 12 14:57 logging.d

Hope this helps!

1 Like

@scorwin I wanted to add that I was testing it on Amazon Linux 2. As far as I know only Amazon Linux 2 is supported, so the previous version might have issues.

You could just use a Log forwarder as a stand-alone (expected FluentBit):

I hope you find this information helpful.

Thanks @kspikowski. I created ticket for our infrastructure team to try following your recommended approach. The ticket should get scheduled today and then done in the next two weeks. Hopefully that will resolve the issue.

If it doesn’t work, at least I’ll have something to support the need to build out newer servers…