Newrelic Infra agent error "can't save submission time"

Hi everyone

Running the Newrelic infra agent on a couple of Ubuntu 18 servers and getting a lot of the below errors.

level=error msg="can't save submission time" component=PatchSender entityKey= error="open /var/db/newrelic-infra/data/last_success: is a directory"

These are EC2 instances in AWS, wondering if the file found in /var/db/newrelic-infra/data/last_success is using the id of the instance i.e. i-8018374938 but I have changed the hostnames of the server itself?

Infra agent version is 1.12.2

Thanks in advanced

1 Like

Hi @james.ashworth1,

That error is indicating last_success is a directory, where it should just be a plain file with content like the following:

2020-08-13T21:19:22.005620254-07:00 

I can’t explain why it’s a directory on your system; I would advise renaming it and re-installing the agent package to address those errors.

Hi @sellefson

Thanks for that I will test that out. So just to confirm last_success should be an actual file? Apposed to it being a directory containing a file named with the instance id i.e. i-8018374938?

Is there anything at a config level that might be making it a directory instead of a file?

At the moment we aren’t doing anything special with the install or anything to over the top with the config files just setting the display_name.

We are automating the install with Puppet (not the puppet module but just a simple slimmed down version) and we eventually be deploying this to 40+ servers so it would be good to figure out the root cause of why its doing this before rolling it out.

Please let me know if you need anything from me.

Regards

Hi @sellefson

Just to add to this, I did what you suggested and got the following error:

level=warning msg="failed when retrieve last submission time" component=PatchSender entityKey=i-00dd2178ec623d0c6 error="open /var/db/newrelic-infra/data/last_success/i-00dd2178ec623d0c6: not a directory"

Hi @james.ashworth1,

Thanks for getting back to me! I spent some time reviewing what the agent uses that directory for, and it appears it is designed to make that a directory and store a timestamp of its latest successful inventory POST to the API.

As a next step; could you try stopping the agent, removing its data directory and then starting it back up again to let the agent fully initialize its local cache?

systemctl stop newrelic-infra
rm -rf /var/db/newrelic-infra/data
systemctl start newrelic-infra

Hi @sellefson

Did as you suggested and to start with everything looked ok but after around 1-2m after starting the agent the error came back

root@staging01:/home/ubuntu# service newrelic-infra status
 newrelic-infra.service - New Relic Infrastructure Agent
 Loaded: loaded (/etc/systemd/system/newrelic-infra.service; enabled; vendor preset: enabled)
 Active: active (running) since Mon 2020-08-31 08:25:11 AEST; 1min 59s ago
 Main PID: 14200 (newrelic-infra-)
 Tasks: 18 (limit: 4396)
 CGroup: /system.slice/newrelic-infra.service
       ├─14200 /usr/bin/newrelic-infra-service
       └─14213 /usr/bin/newrelic-infra

Aug 31 08:25:13 xxxx.xx.xx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:25:13+10:00" level=info msg="Agent plugin" plugin=system/network_interfaces
Aug 31 08:25:13 xxxx.xx.xx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:25:13+10:00" level=info msg="Agent plugin" plugin=kernel/sysctl
Aug 31 08:25:13 xxxx.xx.xx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:25:13+10:00" level=info msg="Agent plugin" plugin=kernel/modules
Aug 31 08:25:13 xxxx.xx.xx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:25:13+10:00" level=info msg="Agent plugin" plugin=services/pidfile
Aug 31 08:25:13 xxxx.xx.xx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:25:13+10:00" level=info msg="Agent plugin" plugin=config/sshd
Aug 31 08:25:13 xxxx.xx.xx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:25:13+10:00" level=info msg="Agent plugin" plugin=packages/dpkg
Aug 31 08:25:13 xxxx.xx.xx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:25:13+10:00" level=info msg="Agent plugin" plugin=config/selinux
Aug 31 08:25:13 xxxx.xx.xx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:25:13+10:00" level=info msg="Agent plugin" plugin=metadata/cloud_security_groups
Aug 31 08:25:14 xxxx.xx.xx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:25:14+10:00" level=info msg="connect got id" agent-guid=MjExNzgxNnxJTkZSQXxOQXw3ODY2NzQ5NDE2MDUyMDkwNzQ2 agent-id=7866749416052090746 component=IdentityConnectService
Aug 31 08:26:04 xxxx.xx.xx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:26:04+10:00" level=error msg="can't save submission time" component=PatchSender entityKey= error="open /var/db/newrelic-infra/data/last_success: is a directory"

I then did a cat of this file /var/db/newrelic-infra/data/last_success/i-00dd2178ec623d0c6 which outputs 2020-08-31T08:29:56.4282839+10:00

The error seems to happen roughly every minute

Aug 31 08:26:04 xxxxx.xxxx.xxxxx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:26:04+10:00" level=error msg="can't save submission time" component=PatchSender entityKey= error="open /var/db/newrelic-infra/data/last_success: is a directory"
Aug 31 08:27:45 xxxx.xx.xxxx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:27:45+10:00" level=error msg="can't save submission time" component=PatchSender entityKey= error="open /var/db/newrelic-infra/data/last_success: is a directory"
Aug 31 08:28:55 xxxx.xx.xx.com.au newrelic-infra-service[14200]: time="2020-08-31T08:28:55+10:00" level=error msg="can't save submission time" component=PatchSender entityKey= error="open /var/db/newrelic-infra/data/last_success: is a directory"

@sellefson just adding some more info with this, it does look like that the file is being updated as it should be see below:

root@xxxx:/var/db/newrelic-infra/data/last_success# cat i-00dd2178ec623d0c6
2020-09-04T07:48:47.542991179+10:00
root@xxxx:/var/db/newrelic-infra/data/last_success# cat i-00dd2178ec623d0c6
2020-09-04T07:49:17.797230685+10:100:
root@xxxx:/var/db/newrelic-infra/data/last_success# cat i-00dd2178ec623d0c6
2020-09-04T07:50:28.275890115+10:00
root@xxx:/var/db/newrelic-infra/data/last_success# cat i-00dd2178ec623d0c6
2020-09-04T07:50:28.275890115+10:00

Hello @james.ashworth1,

Thanks for trying that! I would recommend uninstalling the agent at this point and re-installing.

Hi @peraut

Is this because there was a new agent released on Sept 3 that fixes this bug:

https://docs.newrelic.com/docs/release-notes/infrastructure-release-notes/infrastructure-agent-release-notes

Fixed

  • #75 Fixed a bug that produced this error message can’t save submission time when incorrectly trying to save submission time for the host entity that was not required.

The above version is installed (1.12.4) and I can confirm that the error is not longer present.

Hey @james.ashworth1 - Great to see you! :smile:

That’s probably why it worked for you, but we couldn’t confirm unless we did some additional digging with you. Are you good to go now or do you need some more context?

Hey @hross great to see you as well :slight_smile: So far everything is looking good now thanks :slight_smile:

Thanks to the Newrelic team on getting into it and applying the fix so quickly.

2 Likes

Thank you @james.ashworth1 for confirming all is well now. I’m going to go ahead and mark as resolved :slight_smile: