Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Newrelic-daemon not loading


#2

Hi @chrsmyrs

Can you try

loglevel = “verbosedebug”

debug should work though but verbosedebug should result in very verbose logging.

Signal 15 is a SIGTERM which generally means human execution signalled the program to end.

When was that signal 15 in context of the start command? It would be interesting if it was before or after the call. To know when it’s occuring.


#3

I have changed to verbosedebug and it still logs nothing. Not sure it’s even getting as far as reading the config file. In fact I changed the log file destination and this new file was not created. Sometimes it doesn’t even start getting stuck at Starting newrelic-daemon (via systemctl)…

The signal 15 was a manual termination. Before this problem it was loading but not working, so I changed the config to external loading (since I have PHP running in multiple ways).


#4

Hi @chrsmyrs

OK so first thing to check now

Does /etc/newrelic/newrelic.cfg exist or is it still the newrelic.cfg.template file name. Hopefully it’s still a template.

If it’s still a template you are in automatic startup mode where the daemon will start with the start of PHP. This actually prevents the use of start and stop of the daemon and may be the reason there hasn’t been a working start/stop of the daemon.

systemctl stop newrelic-daemon

Stop that daemon clean. The restart PHP. If you’re using mod_php then just restart apache

systemctl restart apache2 or systemctl restart httpd

If you’re using FPM, you need to restart the PHP-FPM service.

systemctl restart PHP-FPM

After restarting PHP, our agent should have started, causing a restart of our daemon. Which should be checkable via

systemctl status newrelic-daemon

Can you let me know if this starts your daemon normally.


PHP Agent reporting randomly
#5

Hi @acuffe

Thing is I run it with mod_php, PHP-FPM and FCGI for different purposes. That’s why I wanted to start it externally.

Therefore I have newrelic.cfg loaded with verbose debug line set. Even if i remove this so allow automatic startup, and then restart everything, I get no luck, see below…

Last login: Mon Aug 10 20:23:05 2015 from 2.31.15.174
[root@firepixel ~]# mv /etc/newrelic/newrelic.cfg /etc/newrelic/newrelic.cfg.bak                    
[root@firepixel ~]# systemctl stop newrelic-daemon
[root@firepixel ~]# systemctl restart httpd
[root@firepixel ~]# systemctl restart php-fpm
[root@firepixel ~]# systemctl status newrelic-daemon
newrelic-daemon.service - SYSV: Starts and stops the New Relic Proxy Daemon.
   Loaded: loaded (/etc/rc.d/init.d/newrelic-daemon)
   Active: inactive (dead) since Wed 2015-08-12 07:47:32 BST; 1min 14s ago
  Process: 29941 ExecStop=/etc/rc.d/init.d/newrelic-daemon stop (code=killed, signal=TERM)
  Process: 4264 ExecStart=/etc/rc.d/init.d/newrelic-daemon start (code=exited, status=0/SUCCESS)

Aug 06 16:20:46 firepixel.co.uk systemd[1]: Starting SYSV: Starts and stops the New Relic Prox......
Aug 06 16:20:46 firepixel.co.uk systemd[1]: Started SYSV: Starts and stops the New Relic Proxy...n..
Aug 12 07:47:32 firepixel.co.uk systemd[1]: Stopping SYSV: Starts and stops the New Relic Prox......
Aug 12 07:47:32 firepixel.co.uk newrelic-daemon[29941]: Stopping New Relic Daemon (newrelic-dae.....
Aug 12 07:47:32 firepixel.co.uk systemd[1]: Stopped SYSV: Starts and stops the New Relic Proxy...n..
Hint: Some lines were ellipsized, use -l to show in full.
[root@firepixel ~]# ps -ef | grep newrelic-daemon
root     30099 29877  0 07:49 pts/0    00:00:00 grep --color=auto newrelic-daemon
[root@firepixel ~]# cat /var/log/newrelic/newrelic-daemon.log
[root@firepixel ~]#

Changing it back gives me this…

[root@firepixel ~]# mv /etc/newrelic/newrelic.cfg.bak /etc/newrelic/newrelic.cfg
[root@firepixel ~]# systemctl restart newrelic-daemon
[root@firepixel ~]# systemctl status newrelic-daemon
newrelic-daemon.service - SYSV: Starts and stops the New Relic Proxy Daemon.
   Loaded: loaded (/etc/rc.d/init.d/newrelic-daemon)
   Active: active (exited) since Wed 2015-08-12 07:51:10 BST; 6s ago
  Process: 29941 ExecStop=/etc/rc.d/init.d/newrelic-daemon stop (code=killed, signal=TERM)
  Process: 30149 ExecStart=/etc/rc.d/init.d/newrelic-daemon start (code=exited, status=0/SUCCESS)

Aug 12 07:51:10 firepixel.co.uk systemd[1]: Starting SYSV: Starts and stops the New Relic Prox......
Aug 12 07:51:10 firepixel.co.uk systemd[1]: Started SYSV: Starts and stops the New Relic Proxy...n..
Hint: Some lines were ellipsized, use -l to show in full.
[root@firepixel ~]# ps -ef | grep newrelic-daemon
root     30169 29877  0 07:51 pts/0    00:00:00 grep --color=auto newrelic-daemon

#6

Hi @chrsmyrs

That’s extremely unusual, sounds like we could do with a big of logging and additional information about your setup, so I’m going to create a support ticket for you from this. Lets get this sorted.


#7

I am having exactly (sounds like the same) issue.

newrelic-repo-5-3.noarch.rpm
Centos 7.1.1503
PHP 5.6.12
Apache 2.4.6

Errors:

# systemctl status newrelic-daemon
newrelic-daemon.service - SYSV: Starts and stops the New Relic Proxy Daemon.
   Loaded: loaded (/etc/rc.d/init.d/newrelic-daemon)
   Active: inactive (dead) since Wed 2015-08-19 03:41:47 ACST; 37s ago
  Process: 11289 ExecStop=/etc/rc.d/init.d/newrelic-daemon stop (code=killed, signal=TERM)

Aug 19 03:37:41 www.<REMOVED> systemd[1]: Starting SYSV: Starts and stops the New Relic Proxy Daemon....
Aug 19 03:37:41 www.<REMOVED> systemd[1]: Started SYSV: Starts and stops the New Relic Proxy Daemon..
Aug 19 03:41:47 www.<REMOVED> systemd[1]: Stopping SYSV: Starts and stops the New Relic Proxy Daemon....
Aug 19 03:41:47 www.<REMOVED> newrelic-daemon[11289]: Stopping New Relic Daemon (newrelic-daemon) ...
Aug 19 03:41:47 www.<REMOVED> systemd[1]: Stopped SYSV: Starts and stops the New Relic Proxy Daemon..

How do I create a support ticket?


#8

Hello,

Please open a support ticket by visiting https://support.newrelic.com/home and we will be happy to look into this issue as soon as possible.


#9

Having the same issue - CentOS 7.1. I’ve tried agent mode and external, unix sockets and TCP ports. Nothing is working reliably yet. I have a ticket in the works. Certainly has to do with systemd … I just haven’t found a way around this yet.


#10

@tzultacah

I’m working with you on that ticket so we’ll continue to investigate there. Hopefully we’ll find a few options for you.


#11

Hi,

I have also these problems since update to 5.x if I remember right.
The problem: every system start / manual call on Debian Jessie based systems with systemctl start newrelic-daemon the init-script is run but notifies always New Relic Daemon: newrelic-daemon already running without really started/starting the daemon.
And we need this because we running PHP as CLI by supervisor/gearman-worker.

Since I had now time to debugging here the cause/patch for this problem.
Would be nice if you could quick test/fix it for next release because this behavior really disturbs / monitoring is missing.

For debugging of the script I hacked the 1st line to #!/bin/bash -x so I got the script run output to journalctl log:

Dec 12 18:46:58 test-offv-01 systemd[1]: Starting LSB: The New Relic Proxy Daemon...
-- Subject: Unit newrelic-daemon.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit newrelic-daemon.service has begun starting up.
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + LANG=
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + LABEL=newrelic-daemon
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + NAME=newrelic-daemon
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + DESC='New Relic Daemon'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: ++ id -u
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + id=0
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' 0 '!=' 0 ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + ulimit -n 2048
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -f /lib/lsb/init-function ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + nrdaemon=
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + cfgfile=
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -f /etc/default/newrelic-daemon ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + . /etc/default/newrelic-daemon
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: ++ cfgfile=
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: ++ nrdaemon=
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -z '' ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -x /usr/bin/newrelic-daemon ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + nrdaemon=/usr/bin/newrelic-daemon
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -z '' ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -f /etc/newrelic/newrelic.cfg ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + cfgfile=/etc/newrelic/newrelic.cfg
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -z /etc/newrelic/newrelic.cfg ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -f /etc/newrelic/newrelic.cfg ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + pidfile=
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -n /etc/newrelic/newrelic.cfg -a -f /etc/newrelic/newrelic.cfg ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: ++ sed -n -e 's/^[         ]*pidfile[         ]*=[         ]*//p' -e 's/[         ]*$//' /etc/newrelic/newrelic.cfg
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + pidfile=
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + pidarg=
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -z '' ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -d /var/run ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + pidfile=/var/run/newrelic-daemon.pid
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -n /var/run/newrelic-daemon.pid ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + pidarg=' --pidfile /var/run/newrelic-daemon.pid'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + cfgarg=
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -n /etc/newrelic/newrelic.cfg -a -f /etc/newrelic/newrelic.cfg ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + cfgarg=' -c /etc/newrelic/newrelic.cfg'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + nrdaemonopts=' -c /etc/newrelic/newrelic.cfg --pidfile /var/run/newrelic-daemon.pid'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + DODTIME=15
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + case "$1" in
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + start
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + running
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -n /var/run/newrelic-daemon.pid -a -f /var/run/newrelic-daemon.pid ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: ++ ps -ef
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: ++ sed -e /grep/d
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: ++ grep '/newrelic-daemon '
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: ++ awk '{if ($3 == 1) {print $2;}}'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + maybepid=5341
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -z 5341 ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + return 0
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + '[' -z '' -a -z '' ']'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + echo 'New Relic Daemon: newrelic-daemon already running'
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: New Relic Daemon: newrelic-daemon already running
Dec 12 18:46:58 test-offv-01 newrelic-daemon[5341]: + exit 0
Dec 12 18:46:58 test-offv-01 systemd[1]: Started LSB: The New Relic Proxy Daemon.
-- Subject: Unit newrelic-daemon.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit newrelic-daemon.service has finished starting up.
--
-- The start-up result is done.

As you can see the script finds it’s own pid maybepid=5341 as nice mentioned fallback routine !

There comes the question why your devs are not using the default routine (I guess because they aren’t sysops :wink: )…:

Here the package wich is base setup for distribution (and I think also for other Disitributions like CentOS:

root@test-offv-01:~# dpkg -S /bin/pidof
sysvinit-utils: /bin/pidof

and it’s correct notify about the run state:

root@test-offv-01:~# pidof /usr/bin/newrelic-daemon
root@test-offv-01:~#

So let’s patch:

diff -p /etc/init.d/newrelic-daemon.orig /etc/init.d/newrelic-daemon
*** /etc/init.d/newrelic-daemon.orig    2016-09-14 23:38:07.000000000 +0000
--- /etc/init.d/newrelic-daemon 2016-12-12 18:57:51.785570562 +0000
*************** running()
*** 127,133 ****
      running_pid "${pid}" "${nrdaemon}" || return 1
      return 0
    else
!     maybepid=`ps -ef | grep '/newrelic-daemon ' | sed -e '/grep/d' | awk '{if ($3 == 1) {print $2;}}'`
      if [ -z "${maybepid}" ]; then
        return 1
      else
--- 127,133 ----
      running_pid "${pid}" "${nrdaemon}" || return 1
      return 0
    else
!     maybepid=`pidof "${nrdaemon}"`
      if [ -z "${maybepid}" ]; then
        return 1
      else

and test :wink:

...
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + case "$1" in
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + start
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + running
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + '[' -n /var/run/newrelic-daemon.pid -a -f /var/run/newrelic-daemon.pid ']'
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: ++ pidof /usr/bin/newrelic-daemon
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + maybepid=
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + '[' -z '' ']'
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + return 1
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + '[' '!' -f /etc/newrelic/newrelic.cfg ']'
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + '[' -z /usr/bin/newrelic-daemon ']'
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + '[' '!' -x /usr/bin/newrelic-daemon ']'
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + '[' -z '' -a -z '' ']'
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + log_begin_msg 'Starting New Relic Daemon: newrelic-daemon'
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + echo -n 'Starting New Relic Daemon: newrelic-daemon'
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: Starting New Relic Daemon: newrelic-daemon+ RETVAL=0
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + /usr/bin/newrelic-daemon -c /etc/newrelic/newrelic.cfg --pidfile /var/run/newrelic-daemon.pid
Dec 12 18:58:48 test-offv-01 newrelic-daemon[8257]: + sleep 1
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + '[' x0 = x0 ']'
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + running
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + '[' -n /var/run/newrelic-daemon.pid -a -f /var/run/newrelic-daemon.pid ']'
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: ++ cat /var/run/newrelic-daemon.pid
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + pid=8277
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + running_pid 8277 /usr/bin/newrelic-daemon
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + pid=8277
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + name=/usr/bin/newrelic-daemon
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + '[' -z 8277 ']'
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + '[' '!' -d /proc/8277 ']'
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + '[' '!' -f /proc/8277/cmdline ']'
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: ++ cat /proc/8277/cmdline
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: ++ tr '\000' '\n'
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: ++ head -n 1
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: ++ cut -d : -f 1
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + cmd=/usr/bin/newrelic-daemon
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + '[' /usr/bin/newrelic-daemon = /usr/bin/newrelic-daemon ']'
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + return 0
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + return 0
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + RETVAL=0
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + '[' -z '' -a -z '' ']'
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + log_end_msg 0
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + '[' 0 -eq 0 ']'
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + echo ' OK'
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: OK
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + return 0
Dec 12 18:58:49 test-offv-01 newrelic-daemon[8257]: + exit 0
Dec 12 18:58:49 test-offv-01 systemd[1]: Started LSB: The New Relic Proxy Daemon.
-- Subject: Unit newrelic-daemon.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit newrelic-daemon.service has finished starting up.
--
-- The start-up result is done.

:wink:

Bests

Reiner

PS/EDIT: the pidof ... should be something added like | cut -d " " -f 2 to get the parent process pid only. if you need it for killing/checking.


#12

Hey @reiner, thanks for the feedback, and the patch suggestion! I’ve gone ahead and forwarded this along to the product development team to take a look into, and I’ll send you an update if one becomes available.

Thanks!


#13

it is stiil actual problem. The patch is correct. Please fix it


#14

This issue is now more than two years old… Any news?


#15

This issue still exists, and the suggested fix is still valid. Could you guys please fix this?


#16

Still existing in 8.6.0.238 (debian) - any efforts of fixing that issue finally?

Regards, Ingo


#17

Is @reiner fix released already?


#18

Last month my saltstack patch state was running without error on new setup instances so I guess there is no fix yet…

So happy birthday to this patch, hope you have a nice birtday :wink:


#19

Happy Birthday indeed to your patch, @reiner ! I am sure it’s saved the day on many occasions these past four years. :blush: :cake:


#20

Perhaps we could get it actually fixed in the package rather than congratulating a user for their fix 4 years later…


#21

@ray.ward - I can assure you we do have this reported with our PHP team. I don’t have any details on a projected fix date, but the PHP agent team is aware of this, and I’ll be sure to add your +1 to the internal report.