Logs not showing up when using New Relic's Infrastructure Agent on Amazon Linux

I follow all the instructions to set up the infrastructure agent on an Amazon Linux machine. It started to report all instance metrics which was great.
Then I went ahead to enable logging, by adding configuring the log file that should be sent to New Relic. But even after half an hour, the logs haven’t shown up.

New Relic Agent Version: 1.11.22
Logs enabled for account: Yes

After enabling logging for the agent, I keep seeing this error there.

level=error msg=“Error occurred while handling the process” component=integrations.Supervisor error=“exit status 127” process=log-forwarder

The log configuration file inside logging.d:

logs:
  - name: testLog
    file: /var/log/test.log

The log file exists and shows data being added as well when I tail it.

Any ideas anyone where I might be going wrong this?

Hi @ankit.jaggi,

The log error snippet you shared indicated that systemd may not be the init system being used with your Linux distro.

You can check this by running the command below:

pidof systemd

It should return a PID.

In addition, one library that is required when running systemd is locate libsystemd.so.0 . Can you please run the command below to see if this library exists:

locate libsystemd.so.0

If you receive an error similar to /var/lib/mlocate/mlocate.db': No such file or directory you can try running:

sudo updatedb 

Followed by:

locate libsystemd.so.0

We have attempted to reproduce your issue in our environment by spinning up a fresh Amazon Linux 2 instance and this comes packaged with systemd and the required library.

Can I ask if you’re using a custom AMI that doesn’t include systemd?

I just want to remind you that this feature is still in Beta so we may discover some edge cases before a finalized version is released as GA.

Regards,

I am also running into the same issue

We are also using Amazon Linux 2 instance (AMI ID: ami-0aaac1a43dea04780) and can verify that this instance does not have libsystemd.so.0 pre-packaged

Hi @ckim,

Running

sudo updatedb 

Followed by:

locate libsystemd.so.0

Should fix the issue for now. Thanks for your input.

sudo updatedb does not work out of the box in these instances either – I had to also manually install mlocate using yum install mlocate

Also,
I followed the steps you gave and ran

  1. sudo updatedb
  2. locate libsystemd.so.0

Still, pidof systemd does not return a PID and I’m still seeing following error from the logs:

...
INFO[0000] Agent plugin                                  plugin=metadata/cloud_security_groups
INFO[0000] connect got id                                agent-guid=FILTERED agent-id=FILTERED component=IdentityConnectService
ERRO[0001] Error occurred while handling the process     component=integrations.Supervisor error="exit status 127" process=log-forwarder
ERRO[0002] Error occurred while handling the process     component=integrations.Supervisor error="exit status 127" process=log-forwarder
ERRO[0003] Error occurred while handling the process     component=integrations.Supervisor error="exit status 127" process=log-forwarder
ERRO[0006] Error occurred while handling the process     component=integrations.Supervisor error="exit status 127" process=log-forwarder
ERRO[0011] Error occurred while handling the process     component=integrations.Supervisor error="exit status 127" process=log-forwarder
ERRO[0027] Error occurred while handling the process     component=integrations.Supervisor error="exit status 127" process=log-forwarder
...

Hi @ckim,

I’m going to need some more detailed information and logs to investigate this issue.
I’ll bring you to a ticket in order to exchange that information and i’ll post any findings here, if relevant, once we get to the bottom of it.

Cheers,

The issue here was that the init system start command used by the user was not matching the systemd service manager

For Linux, ensure you use the correct command for your init system. Select start, stop, restart, or status as appropriate:

SystemD (Amazon Linux 2, SLES 12, CentOS 7 or higher, Debian 8 or higher, RHEL 7 or higher, Ubuntu 15.04 or higher):

sudo systemctl <start|stop|restart|status> newrelic-infra
1 Like

I’m having this same issue with CentOS 6.10. I get the same errors as the OP.

pidof systemd returns nothing as well as the locate libsystemd.so.0

for some reason my agent does not start with any init processes (no startup script was made during yum install), I have to start it VIA command-line.

systemctl start newrelic-infra shows systemctl: command not found

Thoughts?

Hi @jeff78, At the moment we only support CentOS - Version 7 or higher for the Infrastructure log forwarding functionality which is why we’re seeing the missing dependency errors here.

Having the same issue here trying to use Infra forwarding.

Using Centos 7 (image from AWS marketplace) and New Relic Infrastructure Agent version: 1.11.45, GoVersion: go1.10.8, GitCommit: f3a73a8090908bb59673134e233b8af081250877

Got in newrelic-infra logs:

level=error msg=“Error occurred while handling the process” component=integrations.Supervisor error=“exit status 127” process=log-forwarder

Config file /etc/newrelic-infra/logging.d/logs.yml is

logs:
  - name: "Some-name"
    file: /tmp/test.log
#(empty line)

Config file ownership - root:root. Test log file has few lines with random text, ownership is ‘centos:centos’.

Used @nogueir’s solution: systemctl to restart. Same results.

Hey @iyeremenko, The log error snippet indicates that systemd may not be the init system being used with your Linux distribution.

Could you share the output of

sudo systemctl status newrelic-infra
1 Like

For anyone using a Windows system running into this problem with their New Relic agent and getting “exit status 3221225781”, you need to install Microsoft Visual C++. After I installed it on my machine the error disappeared.

time="2020-06-25T07:04:50-04:00" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 3221225781" process=log-forwarder
time="2020-06-25T07:04:51-04:00" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 3221225781" process=log-forwarder
time="2020-06-25T07:04:53-04:00" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 3221225781" process=log-forwarder
time="2020-06-25T07:04:55-04:00" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 3221225781" process=log-forwarder
time="2020-06-25T07:05:01-04:00" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 3221225781" process=log-forwarder
time="2020-06-25T07:05:03-04:00" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 3221225781" process=log-forwarder
time="2020-06-25T07:05:09-04:00" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 3221225781" process=log-forwarder
time="2020-06-25T07:05:16-04:00" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 3221225781" process=log-forwarder
time="2020-06-25T07:05:55-04:00" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 3221225781" process=log-forwarder
time="2020-06-25T07:08:08-04:00" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 3221225781" process=log-forwarder
1 Like

Thanks for sharing that @ynieves :slight_smile:

Hi nogueir,

I’m with same problem, on Amazon Linux 2018.03 don’t has systemd. Does the one way is upgrade my system?

cat /etc/os-release
NAME=“Amazon Linux AMI”
VERSION=“2018.03”
ID=“amzn”
ID_LIKE=“rhel fedora”
VERSION_ID=“2018.03”
PRETTY_NAME=“Amazon Linux AMI 2018.03”
ANSI_COLOR=“0;33”
CPE_NAME=“cpe:/o:amazon:linux:2018.03:ga”
HOME_URL=“http://aws.amazon.com/amazon-linux-ami/

Jul 31 14:22:21 kernel: [26472.811806] [4232(newrelic-infra)]: gsch_scan(inode=398483,type=1,flags=0x0) - interrupted & wait(timeout=250)
Jul 31 14:22:21 kernel: [26472.819328] [4232(newrelic-infra)]: gsch_scan(inode=398483,type=1,flags=0x0) - interrupted & wait: done
Jul 31 14:23:13 newrelic-infra: time=“2020-07-31T14:23:13-03:00” level=error msg=“Error occurred while handling the process” component=integrations.Supervisor error=“exit 127”
Jul 31 14:28:13 newrelic-infra: time=“2020-07-31T14:28:13-03:00” level=error msg=“Error occurred while handling the process” component=integrations.Supervisor error=“exit 127”
Jul 31 14:33:13 newrelic-infra: time=“2020-07-31T14:33:13-03:00” level=error msg=“Error occurred while handling the process” component=integrations.Supervisor error=“exit 127”

Hi @rodrigo.fedosi, At the moment we only support Amazon Linux 2 (versus the original Amazon Linux AMI) for the Infrastructure log forwarding functionality.

Unfortunately, upgrading to Amazon Linux 2 is the best option and only recommendation here. The Amazon Linux 1 was obsolete and there would be a lot of package error / required to use systemd on it and there’s no guarantee that it would be possible to get things working in the end.

I hope that makes sense! This is essentially just caused by missing dependencies on the Amazon Linux 1 image.