Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

PHP agent 9.0.0.242-1 Segmentation fault


#1

Please paste the [permalink][1] to the page in question below:

N/A

Please share your agent version and other relevant versions below:

9.0.0.242-1

Please share your question/describe your issue below. Include any screenshots that may help us understand your question:

Our New Relic PHP agent was just updated overnight and I’ve started noticing a lot of these segmentation errors (an example below). This also somehow slowed down our Laravel queue workers by a lot, the jobs kept expiring and doing nothing. I’ve reverted back to the PHP agent 8.7 for now.

Here’s an example error I’m getting in my logs (I cannot upload a text file since I’m new here…):

*** Error in `/usr/bin/php': realloc(): invalid old size: 0x00007f0d5cfe9170 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f754)[0x7f0d65b06754]
/lib64/libc.so.6(+0x83d58)[0x7f0d65b0ad58]
/lib64/libc.so.6(realloc+0x1d2)[0x7f0d65b0c9b2]
/usr/lib64/php/modules/newrelic.so(+0x62159)[0x7f0d505b5159]
/usr/lib64/php/modules/newrelic.so(+0x4b2fe)[0x7f0d5059e2fe]
/usr/lib64/php/modules/newrelic.so(+0x243b8)[0x7f0d505773b8]
/usr/lib64/php/modules/newrelic.so(+0x24a82)[0x7f0d50577a82]
/usr/bin/php(zend_call_function+0x7fb)[0x55e52d2483eb]
/usr/bin/php(zend_call_method+0x1df)[0x55e52d27793f]
/usr/bin/php(zend_objects_destroy_object+0xbc)[0x55e52d292e1c]
/usr/bin/php(zend_objects_store_del+0x108)[0x55e52d298238]
/usr/bin/php(zend_array_destroy+0x25d)[0x55e52d26a5ad]
/usr/bin/php(destroy_op_array+0x64)[0x55e52d24b704]
/usr/bin/php(+0x30a7ef)[0x55e52d2837ef]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(zend_array_destroy+0x25d)[0x55e52d26a5ad]
/usr/bin/php(destroy_op_array+0x64)[0x55e52d24b704]
/usr/bin/php(+0x30a7ef)[0x55e52d2837ef]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(zend_array_destroy+0x25d)[0x55e52d26a5ad]
/usr/bin/php(destroy_op_array+0x64)[0x55e52d24b704]
/usr/bin/php(+0x30a7ef)[0x55e52d2837ef]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(zend_object_std_dtor+0x85)[0x55e52d292cc5]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(zend_array_destroy+0x25d)[0x55e52d26a5ad]
/usr/bin/php(destroy_op_array+0x64)[0x55e52d24b704]
/usr/bin/php(+0x30a7ef)[0x55e52d2837ef]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(zend_object_std_dtor+0x85)[0x55e52d292cc5]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(zend_array_destroy+0x25d)[0x55e52d26a5ad]
/usr/bin/php(destroy_op_array+0x64)[0x55e52d24b704]
/usr/bin/php(+0x30a7ef)[0x55e52d2837ef]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(zend_object_std_dtor+0x85)[0x55e52d292cc5]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(zend_array_destroy+0x151)[0x55e52d26a4a1]
/usr/bin/php(zend_object_std_dtor+0x85)[0x55e52d292cc5]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(zend_object_std_dtor+0x85)[0x55e52d292cc5]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(zend_object_std_dtor+0x85)[0x55e52d292cc5]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(zend_array_destroy+0x25d)[0x55e52d26a5ad]
/usr/bin/php(destroy_op_array+0x64)[0x55e52d24b704]
/usr/bin/php(+0x30a7ef)[0x55e52d2837ef]
/usr/bin/php(zend_objects_store_del+0x7c)[0x55e52d2981ac]
/usr/bin/php(+0x38898c)[0x55e52d30198c]
/usr/bin/php(execute_ex+0x325a)[0x55e52d305cba]
/usr/lib64/php/modules/newrelic.so(+0x3e222)[0x7f0d50591222]
/usr/lib64/php/modules/newrelic.so(+0x243ca)[0x7f0d505773ca]
/usr/lib64/php/modules/newrelic.so(+0x24a82)[0x7f0d50577a82]
/usr/bin/php(+0x32476e)[0x55e52d29d76e]
/usr/bin/php(execute_ex+0x44d8)[0x55e52d306f38]
/usr/lib64/php/modules/newrelic.so(+0x3e222)[0x7f0d50591222]
/usr/lib64/php/modules/newrelic.so(+0x243ca)[0x7f0d505773ca]
/usr/lib64/php/modules/newrelic.so(+0x24a82)[0x7f0d50577a82]
/usr/bin/php(+0x32429c)[0x55e52d29d29c]
/usr/bin/php(execute_ex+0x44d8)[0x55e52d306f38]
/usr/lib64/php/modules/newrelic.so(+0x3e222)[0x7f0d50591222]
/usr/lib64/php/modules/newrelic.so(+0x243ca)[0x7f0d505773ca]
/usr/lib64/php/modules/newrelic.so(+0x24a82)[0x7f0d50577a82]
======= Memory map: ========
55e52cf79000-55e52d3ec000 r-xp 00000000 fd:00 79517115                   /usr/bin/php
55e52d5eb000-55e52d662000 r--p 00472000 fd:00 79517115                   /usr/bin/php
55e52d662000-55e52d67d000 rw-p 004e9000 fd:00 79517115                   /usr/bin/php
55e52d67d000-55e52d69a000 rw-p 00000000 00:00 0
55e52d70e000-55e52dd11000 rw-p 00000000 00:00 0                          [heap]
7f0d48000000-7f0d48021000 rw-p 00000000 00:00 0
7f0d48021000-7f0d4c000000 ---p 00000000 00:00 0
7f0d4d304000-7f0d4d32c000 r-xp 00000000 fd:00 134315957                  /usr/lib64/libnsspem.so
7f0d4d32c000-7f0d4d52c000 ---p 00028000 fd:00 134315957                  /usr/lib64/libnsspem.so
7f0d4d52c000-7f0d4d52d000 r--p 00028000 fd:00 134315957                  /usr/lib64/libnsspem.so
7f0d4d52d000-7f0d4d52e000 rw-p 00029000 fd:00 134315957                  /usr/lib64/libnsspem.so
7f0d4d52e000-7f0d4d530000 r-xp 00000000 fd:00 134344530                  /usr/lib64/libnsssysinit.so
7f0d4d530000-7f0d4d72f000 ---p 00002000 fd:00 134344530                  /usr/lib64/libnsssysinit.so
7f0d4d72f000-7f0d4d730000 r--p 00001000 fd:00 134344530                  /usr/lib64/libnsssysinit.so
7f0d4d730000-7f0d4d731000 rw-p 00002000 fd:00 134344530                  /usr/lib64/libnsssysinit.so
7f0d4d731000-7f0d4d7b4000 r-xp 00000000 fd:00 135295443                  /usr/lib64/libfreeblpriv3.so
7f0d4d7b4000-7f0d4d9b3000 ---p 00083000 fd:00 135295443                  /usr/lib64/libfreeblpriv3.so
7f0d4d9b3000-7f0d4d9b5000 r--p 00082000 fd:00 135295443                  /usr/lib64/libfreeblpriv3.so
7f0d4d9b5000-7f0d4d9b6000 rw-p 00084000 fd:00 135295443                  /usr/lib64/libfreeblpriv3.so
7f0d4d9b6000-7f0d4d9ba000 rw-p 00000000 00:00 0
7f0d4d9ba000-7f0d4d9f6000 r-xp 00000000 fd:00 134684993                  /usr/lib64/libsoftokn3.so
7f0d4d9f6000-7f0d4dbf6000 ---p 0003c000 fd:00 134684993                  /usr/lib64/libsoftokn3.so
7f0d4dbf6000-7f0d4dbf7000 r--p 0003c000 fd:00 134684993                  /usr/lib64/libsoftokn3.so
7f0d4dbf7000-7f0d4dbf8000 rw-p 0003d000 fd:00 134684993                  /usr/lib64/libsoftokn3.so
7f0d4dbf8000-7f0d4dbfd000 r-xp 00000000 fd:00 134504655                  /usr/lib64/libnss_dns-2.17.so
7f0d4dbfd000-7f0d4ddfd000 ---p 00005000 fd:00 134504655                  /usr/lib64/libnss_dns-2.17.so
7f0d4ddfd000-7f0d4ddfe000 r--p 00005000 fd:00 134504655                  /usr/lib64/libnss_dns-2.17.so
7f0d4ddfe000-7f0d4ddff000 rw-p 00006000 fd:00 134504655                  /usr/lib64/libnss_dns-2.17.so
7f0d4ddff000-7f0d4de00000 ---p 00000000 00:00 0
7f0d4de00000-7f0d50200000 rw-p 00000000 00:00 0
7f0d50340000-7f0d5034c000 r-xp 00000000 fd:00 134504657                  /usr/lib64/libnss_files-2.17.so
7f0d5034c000-7f0d5054b000 ---p 0000c000 fd:00 134504657                  /usr/lib64/libnss_files-2.17.so
7f0d5054b000-7f0d5054c000 r--p 0000b000 fd:00 134504657                  /usr/lib64/libnss_files-2.17.so
7f0d5054c000-7f0d5054d000 rw-p 0000c000 fd:00 134504657                  /usr/lib64/libnss_files-2.17.so
7f0d5054d000-7f0d50553000 rw-p 00000000 00:00 0
7f0d50553000-7f0d5061a000 r-xp 00000000 fd:00 543342822                  /usr/lib/newrelic-php5/agent/x64/newrelic-20170718.so
7f0d5061a000-7f0d50819000 ---p 000c7000 fd:00 543342822                  /usr/lib/newrelic-php5/agent/x64/newrelic-20170718.so
7f0d50819000-7f0d5081e000 rw-p 000c6000 fd:00 543342822                  /usr/lib/newrelic-php5/agent/x64/newrelic-20170718.so
7f0d5081e000-7f0d50820000 rw-p 00000000 00:00 0
7f0d50820000-7f0d50883000 r-xp 00000000 fd:00 68514852                   /usr/lib64/php/modules/redis.so
7f0d50883000-7f0d50a82000 ---p 00063000 fd:00 68514852                   /usr/lib64/php/modules/redis.so
7f0d50a82000-7f0d50a83000 r--p 00062000 fd:00 68514852                   /usr/lib64/php/modules/redis.so
7f0d50a83000-7f0d50a87000 rw-p 00063000 fd:00 68514852                   /usr/lib64/php/modules/redis.so
7f0d50a87000-7f0d50a88000 rw-p 00000000 00:00 0
7f0d50a88000-7f0d50aa0000 r-xp 00000000 fd:00 134743616                  /usr/lib64/libzip.so.5.0
7f0d50aa0000-7f0d50c9f000 ---p 00018000 fd:00 134743616                  /usr/lib64/libzip.so.5.0
7f0d50c9f000-7f0d50ca0000 r--p 00017000 fd:00 134743616                  /usr/lib64/libzip.so.5.0
7f0d50ca0000-7f0d50ca1000 rw-p 00018000 fd:00 134743616                  /usr/lib64/libzip.so.5.0
7f0d50ca1000-7f0d50cae000 r-xp 00000000 fd:00 67690507                   /usr/lib64/php/modules/zip.so
7f0d50cae000-7f0d50eae000 ---p 0000d000 fd:00 67690507                   /usr/lib64/php/modules/zip.so
7f0d50eae000-7f0d50eb0000 r--p 0000d000 fd:00 67690507                   /usr/lib64/php/modules/zip.so
7f0d50eb0000-7f0d50eb1000 rw-p 0000f000 fd:00 67690507                   /usr/lib64/php/modules/zip.so
7f0d50eb1000-7f0d50ed2000 r-xp 00000000 fd:00 68508520                   /usr/lib64/php/modules/igbinary.so
7f0d50ed2000-7f0d510d1000 ---p 00021000 fd:00 68508520                   /usr/lib64/php/modules/igbinary.so
7f0d510d1000-7f0d510d2000 r--p 00020000 fd:00 68508520                   /usr/lib64/php/modules/igbinary.so
7f0d510d2000-7f0d510d3000 rw-p 00021000 fd:00 68508520                   /usr/lib64/php/modules/igbinary.so
7f0d510d3000-7f0d510da000 r-xp 00000000 fd:00 72910883                   

cut off due to limit…


#2

Seeing the same issues since yesterday on any new instances that are scaled up.

Out of interest, how do you specify the version to install?


#3

First I uninstalled newrelic agent:

sudo yum remove newrelic-php5 newrelic-php5-common

and then installed a specific version like so:

sudo yum install newrelic-php5-8.7.0.242-1

package name, dash, and the version number.
[package-name]-[version]

You can find the available versions by running this:

yum --showduplicates list newrelic-php5

#4

Hey folks! @arunas1 & @mark77 - sorry you’ve had that experience with the new agent - I’ve reached out to the PHP agent team internally. I’ll follow up when I know more. :smiley:


#5

Hey @arunas1 & @mark77 -

I just chat with our PHP Devs - they’re looking into this now as a priority. While they do run rigorous testing with the agent, they did not come across segfaults with v9.0. Currently the advice is that if you experience Segfaults, revert to agent 8.7…


#6

Happening to us too, spent all day trying to figure out what was causing these seg faults…


#7

Hey @dhenry1 - thanks for letting us know this is happening for you too! Our developers are currently reviewing logs sent in by @arunas1 - so we will hopefully know the cause and have a resolution soon.

In the meantime, please revert to PHP agent 8.7.

Thank you!


#8

We noticed the same behaviour. We get segmentation faults when PHP code has CURL requests


#9

Thanks @jonas.galdikas for confirming that - could you use agent 8.7 for now?


#10

Yes, we have downgraded on Friday


#11

Great! Thanks @jonas.galdikas


#12

We experienced the same thing over the weekend, specifically during calls in which cookie values were read.

As it broke our signup stack, it cost the company a significant sum of money.


#13

We’ve also experienced the same thing. Any time anything was attempted to be loaded the whole PHP child process died meaning the stack died any time it was called.

@RyanVeitch New Relic should definitely be withdrawing this from package managers and as an official release. It’s severely broken and we’ve spent hours to find this thread for a solution of rolling back. Not everyone else is as lucky to see this as the solution quick!


#14

Hey @grassenti @theo5

I’m sorry you’ve experienced this issue. To your point, Theo, we are working on getting our development team to remove the agent from package managers.

Please rest assured that we treat any problems introduced by our agents with a high priority. Seg faults among the most impactful of them. Our PHP dev team are working hard right now on reproducing these problems, in order to find the exact root cause and make the necessary changes.

Please do revert to agent 8.7 or lower to get to a more stable agent.


#15

@RyanVeitch — The release of New Relic PHP Agent 9.0 also made our platform seg-fault this weekend. Caused great distress to my team and our customers and consumed a lot of everyone’s time.

Can you revise your docs (e.g., https://docs.newrelic.com/docs/agents/php-agent/frameworks-libraries/aws-elastic-beanstalk-installation-php), so as to stop suggesting that we use the latest version of New Relic Agent? We would have avoided a lot of trouble if we’d specified PHP Agent 8.7 RPMs in our .ebextensions and stuck with 8.7 until 9.0 was verifiably stable.

[Mon Aug 12 00:36:56.148019 2019] [core:notice] [pid 18178] AH00052: child pid 17140 exit signal Segmentation fault (11)
[Mon Aug 12 00:37:01.157273 2019] [core:notice] [pid 18178] AH00052: child pid 16181 exit signal Segmentation fault (11)
*** Error in `/usr/sbin/httpd’: realloc(): invalid pointer: 0x0000560ae9717ee8 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f644)[0x7ff089a9d644]
/lib64/libc.so.6(realloc+0x389)[0x7ff089aa3a59]
/usr/lib64/php/7.1/modules/newrelic.so(+0x62109)[0x7ff06b219109]
/usr/lib64/php/7.1/modules/newrelic.so(+0x4b2ae)[0x7ff06b2022ae]
/usr/lib64/php/7.1/modules/newrelic.so(+0x24368)[0x7ff06b1db368]
/usr/lib64/php/7.1/modules/newrelic.so(+0x24a32)[0x7ff06b1dba32]
/etc/httpd/modules/libphp-7.1.so(+0x2d6725)[0x7ff07d81b725]
/etc/httpd/modules/libphp-7.1.so(execute_ex+0x2b)[0x7ff07d7ca59b]
/usr/lib64/php/7.1/modules/newrelic.so(+0x3e1d2)[0x7ff06b1f51d2]
/usr/lib64/php/7.1/modules/newrelic.so(+0x2437a)[0x7ff06b1db37a]
/usr/lib64/php/7.1/modules/newrelic.so(+0x24a32)[0x7ff06b1dba32]
/etc/httpd/modules/libphp-7.1.so(+0x2d6725)[0x7ff07d81b725]
/etc/httpd/modules/libphp-7.1.so(execute_ex+0x2b)[0x7ff07d7ca59b]
/usr/lib64/php/7.1/modules/newrelic.so(+0x3e1d2)[0x7ff06b1f51d2]
/usr/lib64/php/7.1/modules/newrelic.so(+0x2437a)[0x7ff06b1db37a]
/usr/lib64/php/7.1/modules/newrelic.so(+0x24a32)[0x7ff06b1dba32]
/etc/httpd/modules/libphp-7.1.so(+0x2d6725)[0x7ff07d81b725]
/etc/httpd/modules/libphp-7.1.so(execute_ex+0x2b)[0x7ff07d7ca59b]

#16

We have a total of 5.4K+ PHP 9.0 agents reporting, but saw issues with a handful of customers. We don’t feel that we have fully identified the root cause and don’t want to cause any more potential issues. We have officially rolled back this release as of right now, and hope to replace it soon with an updated version.


#17

Fire drill for us today as well.


#18

Hi all - I would like to point this thread to @jvarney’s post detailing that we have rolled back the PHP agent release for version 9.0.0.242


Further updates will be posted here by Jodee :


#19

more details what I’ve found.

*** Error in `/usr/bin/php': corrupted double-linked list (not small): 0x0000000003a93ba0 ***
*** Error in `/usr/bin/php': free(): invalid size: 0x00007f976f1dcf20 ***
*** Error in `/usr/bin/php': realloc(): invalid old size: 0x00007f87e8f49cf0 ***

laravel-worker_01.txt (50.5 KB)
laravel-worker_03.txt (50.6 KB)
laravel-worker_06.txt (50.4 KB)


#20

Thanks for sharing those details! @ben26 - We’ll have those looked into along with the data from other users experiencing Seg Faults. :slight_smile: