Infrastructure agent failing to deliver logs on AWS Elastic Beanstalk

We’re using New Relic for logging from an Elastic Beanstalk server running Amazon Linux 2. Things have been working fine the past few months, but a few hours ago today, logs stopped getting delivered. I’ve SSH’ed into one of the servers and run:

sudo systemctl status newrelic-infra -l

And this is the output:

May 03 22:42:16 ip-172-31-43-134.us-west-2.compute.internal newrelic-infra-service[3100]: time="2021-05-03T22:42:16Z" level=debug msg="Patch sender found no deltas to send." component=PatchSender entityKey=

May 03 22:42:16 ip-172-31-43-134.us-west-2.compute.internal newrelic-infra-service[3100]: time="2021-05-03T22:42:16Z" level=debug msg="Patch sender found no deltas to send." component=PatchSender entityKey=i-01567c987b5e5d344

May 03 22:42:20 ip-172-31-43-134.us-west-2.compute.internal newrelic-infra-service[3100]: time="2021-05-03T22:42:20Z" level=debug msg="Delta storage." component=SystemSampler elapsedMs=19997 totalReadTime=0 totalReads=0 totalWriteTime=580 totalWrites=124

May 03 22:42:20 ip-172-31-43-134.us-west-2.compute.internal newrelic-infra-service[3100]: time="2021-05-03T22:42:20Z" level=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender key=8607309082720320837 numEvents=2 postCount=2072 timestamps="[2021-05-03 22:42:20 +0000 UTC]"

May 03 22:42:20 ip-172-31-43-134.us-west-2.compute.internal newrelic-infra-service[3100]: time="2021-05-03T22:42:20Z" level=debug msg="Preparing metrics post." component=MetricsIngestSender postCount=2072

May 03 22:42:20 ip-172-31-43-134.us-west-2.compute.internal newrelic-infra-service[3100]: time="2021-05-03T22:42:20Z" level=debug msg="Metrics post succeeded." component=MetricsIngestSender postCount=2072

May 03 22:42:24 ip-172-31-43-134.us-west-2.compute.internal newrelic-infra-service[3100]: time="2021-05-03T22:42:24Z" level=info msg="Temporal verbose log end, restored previous log level" component=Verboser fields.level=info

May 03 22:42:48 ip-172-31-43-134.us-west-2.compute.internal newrelic-infra-service[3100]: time="2021-05-03T22:42:48Z" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 127" process=log-forwarder

May 03 22:47:48 ip-172-31-43-134.us-west-2.compute.internal newrelic-infra-service[3100]: time="2021-05-03T22:47:48Z" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 127" process=log-forwarder

May 03 22:52:48 ip-172-31-43-134.us-west-2.compute.internal newrelic-infra-service[3100]: time="2021-05-03T22:52:48Z" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 127" process=log-forwarder

Can anyone help?

Hey, this is really urgent - we’ve gotten no data for the past few hours. Can someone please advise? I’ve been unsuccessful so far figuring out what that 127 error code means.

Hi @jacob29 - It looks like that host was running Infrastructure Agent v1.17.0 - Any chance you’d updated to that recently?

v1.17.0 has a bug with log forwarding that would have caused the “exit status 127” error you see at the end of the logs you provided. However, we resolved that bug in v1.17.1 (Release notes) if you could try updating the Agent and see it that helps!

Absolutely let us know if you continue to run into any issues though.

We redeployed and may not have locked in an Agent version, so it could have been that. Worth noting that we never found a solution. Thanks.

1 Like