Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Open /tmp/nr-integrations/com.company.custom-golang-metric.json: permission denied

newrelic-infra
infrastructure
on-host-integrations

#1

I’ve followed the guide (here: https://github.com/newrelic/infra-integrations-sdk/blob/master/docs/tutorial.md) to create a custom Infrastructure Integration and when I run “initctl restart newrelic-infra” I get the below verbose logs:

    time="2019-10-31T17:00:26Z" level=info msg="Integration health check starting" instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:00:26Z" level=debug msg="converted relative executable to its absolute path" absolutePath=/var/db/newrelic-infra/custom-integrations/bin/company-custom-golang-metrics instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics relativePath=./bin/company-custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:00:26Z" level=debug msg="running command" args="[/var/db/newrelic-infra/custom-integrations/bin/company-custom-golang-metrics --metrics]" env="[VERBOSE=1 UPPER_BOUND=70 LOWER_BOUND=50 PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin]" instance=custom-golang-metrics integration=com.company.custom-golang-metrics path=/var/db/newrelic-infra/custom-integrations/bin/company-custom-golang-metrics prefix=integration/com.company.custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:00:26Z" level=debug msg="starting sampler" component="Metrics Sender" sampler=SystemSampler
    time="2019-10-31T17:00:26Z" level=debug msg="starting sampler" component="Metrics Sender" sampler=StorageSampler
    time="2019-10-31T17:00:26Z" level=debug msg="starting sampler" component="Metrics Sender" sampler=NFSSampler
    time="2019-10-31T17:00:26Z" level=debug msg="starting sampler" component="Metrics Sender" sampler=NetworkSampler
    time="2019-10-31T17:00:26Z" level=debug msg="starting sampler" component="Metrics Sender" sampler=ProcessSampler
    time="2019-10-31T17:00:26Z" level=debug msg="started sampler routine" component="Metrics Sender" name=ProcessSampler
    time="2019-10-31T17:00:26Z" level=debug msg="started sampler routine" component="Metrics Sender" name=SystemSampler
    time="2019-10-31T17:00:26Z" level=debug msg="started sampler routine" component="Metrics Sender" name=StorageSampler
    time="2019-10-31T17:00:26Z" level=debug msg="started sampler routine" component="Metrics Sender" name=NFSSampler
    time="2019-10-31T17:00:26Z" level=debug msg="started sampler routine" component="Metrics Sender" name=NetworkSampler
    time="2019-10-31T17:00:26Z" level=debug msg="starting harvest" component=Plugins id=metadata/host_aliases
    time="2019-10-31T17:00:26Z" level=error msg="Unable to read sysctl from /proc/sys/fs/protected_fifos, skipping: open /proc/sys/fs/protected_fifos: permission denied" plugin=Sysctl
    time="2019-10-31T17:00:26Z" level=error msg="Unable to read sysctl from /proc/sys/fs/protected_regular, skipping: open /proc/sys/fs/protected_regular: permission denied" plugin=Sysctl
    time="2019-10-31T17:00:26Z" level=error msg="Unable to read sysctl from /proc/sys/net/core/bpf_jit_limit, skipping: open /proc/sys/net/core/bpf_jit_limit: permission denied" plugin=Sysctl
    time="2019-10-31T17:00:26Z" level=warning msg="unable to use upstart" error="exit status 1" plugin=Upstart
    time="2019-10-31T17:00:26Z" level=debug msg="integration returned no output" instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:00:26Z" level=debug msg="Integration command wrote to stderr" instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics stderr="17:00:26.786617 [DEBUG] store file (/tmp/nr-integrations/com.company.custom-golang-metrics.json) is older than 1m0s, skipping loading from disk.\n[ERR] open /tmp/nr-integrations/com.company.custom-golang-metrics.json: permission denied\n" working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:00:26Z" level=info msg="Integration health check finished with some errors" instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations

I thought as it seems to be some kind of permissions issue, I’d change the definition file to include “integration_user: nri-agent” but that generated even stranger issues like “fork/exec /bin/sudo: no such file or directory”.

    time="2019-10-31T17:41:44Z" level=info msg="Integration health check starting" instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:41:44Z" level=debug msg="converted relative executable to its absolute path" absolutePath=/var/db/newrelic-infra/custom-integrations/bin/company-custom-golang-metrics instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics relativePath=./bin/company-custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:41:44Z" level=debug msg="running command" args="[/bin/sudo -E -n -u nri-agent /var/db/newrelic-infra/custom-integrations/bin/company-custom-golang-metrics --metrics]" env="[VERBOSE=1 LOWER_BOUND=50 UPPER_BOUND=70 PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin]" instance=custom-golang-metrics integration=com.company.custom-golang-metrics path=/bin/sudo prefix=integration/com.company.custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:41:44Z" level=error msg="starting data source" error="fork/exec /bin/sudo: no such file or directory" instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:41:44Z" level=error msg="reading stderr" error="read |0: file already closed" instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:41:44Z" level=debug msg="starting sampler" component="Metrics Sender" sampler=SystemSampler
    time="2019-10-31T17:41:44Z" level=debug msg="starting sampler" component="Metrics Sender" sampler=StorageSampler
    time="2019-10-31T17:41:44Z" level=debug msg="starting sampler" component="Metrics Sender" sampler=NFSSampler
    time="2019-10-31T17:41:44Z" level=debug msg="starting sampler" component="Metrics Sender" sampler=NetworkSampler
    time="2019-10-31T17:41:44Z" level=debug msg="starting sampler" component="Metrics Sender" sampler=ProcessSampler
    time="2019-10-31T17:41:44Z" level=debug msg="started sampler routine" component="Metrics Sender" name=ProcessSampler
    time="2019-10-31T17:41:44Z" level=debug msg="Registering inventory for entity" component=Agent entityKey=
    time="2019-10-31T17:41:44Z" level=debug msg="starting harvest" component=Plugins id=metadata/host_aliases
    time="2019-10-31T17:41:44Z" level=warning msg="reading integration output" error="read |0: file already closed" instance=custom-golang-metrics integration=com.company.custom-golang-metrics outputLine="[]" prefix=integration/com.company.custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:41:44Z" level=debug msg="integration returned no output" instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:41:44Z" level=error msg="Integration command failed" error="exec: not started" instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics stderr= working-dir=/var/db/newrelic-infra/custom-integrations
    time="2019-10-31T17:41:44Z" level=info msg="Integration health check finished with some errors" instance=custom-golang-metrics integration=com.company.custom-golang-metrics prefix=integration/com.company.custom-golang-metrics working-dir=/var/db/newrelic-infra/custom-integrations

I assume I’m doing something very wrong, can you please point me in the right direction.


#2

Hi @Kostas.Theo

It looks like you’re running the agent as a non-root user. The permissions error indicates that the user you have the agent running as doesn’t have the rights to modify a file under the /tmp/nr-integrations/ directory:

I would double-check who owns that directory with the following:

ls -lZd /tmp/nr-integrations

It looks like your host doesn’t have sudo installed, which is what agents running as a non-root user with the privileged run mode attempt to use. You can check if it’s installed with which sudo or by looking at your package manager:

yum list sudo
apt-cache show sudo

#3

Hi @sellefson,

Thanks for getting back to me on this thread.

I’ve run the commands you asked for and bellow is the output:

[kostas.theo@i-00cc0d0c7d4a26062 ~]$ ls -lZd /tmp/nr-integrations
drwxr-xr-x root root ?                                /tmp/nr-integrations
[kostas.theo@i-00cc0d0c7d4a26062 ~]$ ls -l /tmp/nr-integrations/
total 4
-rw-r--r-- 1 root root 11 Oct 31 13:14 com.company.custom-golang-metric.json
[kostas.theo@i-00cc0d0c7d4a26062 ~]$ yum list sudo
Loaded plugins: priorities, update-motd, upgrade-helper
Installed Packages
sudo.x86_64               1.8.6p3-29.28.amzn1            @amzn-updates

Especially useful in this case is:

[root@i-00cc0d0c7d4a26062 kostas.theo]# echo $PATH
/sbin:/bin:/usr/sbin:/usr/bin
[root@i-00cc0d0c7d4a26062 kostas.theo]# which sudo
/usr/bin/sudo

The host does indeed have sudo installed. Does the agent need to be so opinionated in specifying the path of sudo in this case? Maybe you have a workaround to suggest in this case?

Furthermore, is the /tmp/nr-integrations directory is created by the agent itself? I would imagine so, in which case, shouldn’t the permissions be already set correctly for the agent to do whatever it needs with that com.company.custom-golang-metric.json file?


#4

I was able to resolve this issue by not specifying integration_user and doing:

chmod 664 /tmp/nr-integrations/com.company.custom-golang-metrics.json

Not sure if this is recommended though. Would still be curious to hear back on the above if that’s okay :slight_smile: