[PHP] Processes stucked on a New Relic related PHP process

Hi,

On May 1st, 3 processes got stucked on one of our linux instances. Here’s the detailed ptrace for one of them:

#0 0x00007f49c96f1a41 in poll () from /lib64/libc.so.6
#1 0x00007f49c7518ee9 in Curl_poll () from /lib64/libcurl.so.4
#2 0x00007f49c75128c0 in curl_multi_wait () from /lib64/libcurl.so.4
#3 0x00007f49c750bcf3 in curl_easy_perform () from /lib64/libcurl.so.4
#4 0x00007f49c77764e8 in zif_curl_exec () from /opt/remi/php71/root/usr/lib64/php/modules/curl.so
#5 0x00007f49b1a32451 in nr_zend_call_old_handler () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_internal_instrument.c:152
#6 0x00007f49b1a33147 in _nr_inner_wrapper_function_curl_exec () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_internal_instrument.c:2187
#7 0x00005640261299dc in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER ()
#8 0x00005640260d9cb3 in execute_ex ()
#9 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#10 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#11 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#12 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#13 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#14 0x00005640260d9cb3 in execute_ex ()
#15 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#16 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#17 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#18 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#19 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#20 0x00005640260d9cb3 in execute_ex ()
#21 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#22 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#23 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#24 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#25 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#26 0x00005640260d9cb3 in execute_ex ()
#27 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#28 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#29 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#30 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#31 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#32 0x00005640260d9cb3 in execute_ex ()
#33 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#34 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#35 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#36 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#37 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#38 0x00005640260d9cb3 in execute_ex ()
#39 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#40 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#41 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#42 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#43 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#44 0x00005640260d9cb3 in execute_ex ()
#45 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#46 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#47 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#48 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#49 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#50 0x00005640260d9cb3 in execute_ex ()
#51 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#52 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#53 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#54 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#55 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#56 0x00005640260d9cb3 in execute_ex ()
#57 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#58 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#59 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#60 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#61 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#62 0x00005640260d9cb3 in execute_ex ()
#63 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#64 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#65 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#66 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#67 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#68 0x00005640260d9cb3 in execute_ex ()
#69 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#70 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#71 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#72 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#73 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#74 0x00005640260d9cb3 in execute_ex ()
#75 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#76 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#77 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#78 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#79 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#80 0x00005640260d9cb3 in execute_ex ()
#81 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#82 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#83 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#84 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#85 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#86 0x00005640260d9cb3 in execute_ex ()
#87 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#88 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#89 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#90 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#91 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#92 0x00005640260d9cb3 in execute_ex ()
#93 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#94 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#95 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#96 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#97 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#98 0x00005640260d9cb3 in execute_ex ()
#99 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#100 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#101 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#102 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#103 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#104 0x00005640260d9cb3 in execute_ex ()
#105 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#106 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#107 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#108 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#109 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#110 0x00005640260d9cb3 in execute_ex ()
#111 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#112 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#113 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#114 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#115 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#116 0x00005640260d9cb3 in execute_ex ()
#117 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#118 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#119 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#120 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#121 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#122 0x00005640260d9cb3 in execute_ex ()
#123 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#124 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#125 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#126 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#127 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#128 0x00005640260d9cb3 in execute_ex ()
#129 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#130 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#131 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#132 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#133 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#134 0x00005640260d9cb3 in execute_ex ()
#135 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#136 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#137 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#138 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#139 0x00005640261299d4 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER ()
#140 0x00005640260d9cb3 in execute_ex ()
#141 0x00007f49b1a43812 in nr_zend_call_orig_execute_special () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_user_instrument.c:70
#142 0x00007f49b1a283ec in nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1194
#143 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#144 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#145 0x0000564026129524 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
#146 0x00005640260d9cb3 in execute_ex ()
#147 0x00007f49b1a282cc in nr_php_execute_file () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:909
#148 nr_php_execute_enabled () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1103
#149 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#150 0x00007f49c8437bee in ?? () from /opt/remi/php71/root/usr/lib64/php/modules/ioncube_loader.so
#151 0x000056402612b6b0 in zend_execute ()
#152 0x000056402609229b in zend_execute_scripts ()
#153 0x000056402602e910 in php_execute_script ()
#154 0x000056402612dbcf in do_cli ()
#155 0x0000564025f05d6c in main ()

The other 2 are the same but shorter. It seems the process enters on a loop on the following line:

#149 0x00007f49b1a28b82 in nr_php_execute () at /home/hudson/workspace/php-release-agent/label/centos5-64-nrcamp/agent/php_execute.c:1337
#150 0x00007f49c8437bee in ?? () from

What should I do to fix it and avoid from it to happen again?

Hello @isidro,

Thank you for posting here in the community. I ran this by one of our developers and we wanted to note that it’s expected that you’ll see a lot of occurrences of nr_php_execute_file in traces as this is what allows the agent to hook into the transaction.

It could be that the loop is caused by curl_multi_wait waiting for external data, which could be taking longer than the timeout alotted. You might try increasing the timeout, but we don’t see any issues that jump out at us in relation to the agent.

If you do see something concerning the agent here, let me know!

Where can I increase the timeout?