[PHP] New Relic completely broken after PHP 8.1 upgrade

Support: Full-Stack Observability (FSO) Agents
, , , ,
#1

I have awoken this morning to one of my New Relic alarms being triggered for >5% error rate. The new relic deamon has stopped on one of the servers.

I am being told by newrelic that the error rate is 100% and all transactions are labeled as unknown. This is following the PHP 8.1 deamon update.

We are running PHP8.1 on Debian server, it’s a Laravel application.

The new relic deamon has been restarted on both servers. They are up and running, an no errors are being thrown.

I highly doubt that this is a coincidence. Am i the only one experiencing this issue ?

Latest logs :

2022-03-14 09:38:20.292 +0000 (18259 18259) warning: daemon connect(fd=16 uds=@newrelic) returned -1 errno=ECONNREFUSED. Failed to connect to the newrelic-daemon. Please make sure that there is a properly configured newrelic-daemon running. For additional assistance, please see: Starting the PHP daemon (advanced) | New Relic Documentation
2022-03-14 09:39:01.420 +0000 (19119 19119) info: attempt daemon connection via ‘@newrelic’
2022-03-14 09:39:01.420 +0000 (19119 19119) info: New Relic 9.19.0.309 (“zomp” - “8536ad58490a”) [daemon=’@newrelic’ php=‘8.1.2’ zts=no sapi=‘cli’ pid=19119 ppid=19108 uid=0 euid=0 gid=0 egid=0 backtrace=yes startup=init os=‘Linux’ rel=‘4.19.0-17-amd64’ mach=‘x86_64’ ver=’#1 SMP Debian 4.19.194-2 (2021-06-21)’ node=‘timbre-01-gos2’]
2022-03-14 09:39:01.488 +0000 (19132 19132) info: attempt daemon connection via ‘@newrelic’
2022-03-14 09:39:01.488 +0000 (19132 19132) info: New Relic 9.19.0.309 (“zomp” - “8536ad58490a”) [daemon=’@newrelic’ php=‘8.1.2’ zts=no sapi=‘cli’ pid=19132 ppid=19108 uid=0 euid=0 gid=0 egid=0 backtrace=yes startup=init os=‘Linux’ rel=‘4.19.0-17-amd64’ mach=‘x86_64’ ver=’#1 SMP Debian 4.19.194-2 (2021-06-21)’ node=‘timbre-01-gos2’]
2022-03-14 09:39:01.550 +0000 (19145 19145) info: attempt daemon connection via ‘@newrelic’
2022-03-14 09:39:01.550 +0000 (19145 19145) info: New Relic 9.19.0.309 (“zomp” - “8536ad58490a”) [daemon=’@newrelic’ php=‘8.1.2’ zts=no sapi=‘cli’ pid=19145 ppid=19108 uid=0 euid=0 gid=0 egid=0 backtrace=yes startup=init os=‘Linux’ rel=‘4.19.0-17-amd64’ mach=‘x86_64’ ver=’#1 SMP Debian 4.19.194-2 (2021-06-21)’ node=‘timbre-01-gos2’]
2022-03-14 09:52:28.664 +0000 (19209 19209) error: TXNDATA failure: len=8180 errno=EPIPE
2022-03-14 09:52:30.232 +0000 (17106 17106) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:30.997 +0000 (19195 19195) error: TXNDATA failure: len=12332 errno=EPIPE
2022-03-14 09:52:31.026 +0000 (19193 19193) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:31.380 +0000 (19216 19216) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:32.692 +0000 (18355 18355) error: TXNDATA failure: len=11732 errno=EPIPE
2022-03-14 09:52:34.194 +0000 (19214 19214) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:37.828 +0000 (19217 19217) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:38.016 +0000 (19194 19194) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:39.934 +0000 (17863 17863) error: TXNDATA failure: len=12332 errno=EPIPE

image
image1256×632 69.2 KB

image
image1622×653 49.4 KB

2 Likes
#2

We are having the same issue after upgrading to php8.1

#3

There are a couple of open issues on the github repo :

  1. https://github.com/newrelic/newrelic-php-agent/issues/387
  2. https://github.com/newrelic/newrelic-php-agent/issues/392

Although they have both gone silent. I was last asked to open a support ticket, but there is no way on the new relic site to do that even when logged in under a pro account. I think i’ve been fobbed off.

#4

@ss-frank @desarrollo7 ,

I hope you are both having a good day.

Please note we read every post we want you to know we take all support questions seriously. Our goal is to always have everyone in the community feel supported.

Unfortunately this support topic is a little outside of my knowledge. Despite this we are looking into this to help find you both a solution.

I will loop in PHP agent expert, they will reply here to support this issue.

I have found some resources that may help you resolve this while we wait for a (whatever area of expertise/category customer needs help with) expert to provide further information.

Apologies for the wait and any frustration felt. Should you have any updates on this please feel free to share them. I hope you have a great day!

#5

Hello @ss-frank and @desarrollo7,

Thank you for your patience as we look further into this issue from our end! First off, I would also like to apologize and acknowledge the frustration you are feeling. I was able to speak to our development team regarding this issue and they were able to confirm the release of a newer version of the PHP agent, 9.21.0.311. They believe this version will address the problem you are seeing and would recommend upgrading to this new version to check if this resolves the issue from your end.

For instructions on how to upgrade the PHP agent, please see the following link:

In the event that you still see this problem after upgrading to version 9.21.0.311, we would definitely want to continue troubleshooting this issue and would love it if you could file a support case to dive in further. We haven’t had any issues with pro accounts being able to file cases, so we believe you should be able to using the following directions:

However, if you are still unsuccessful in being able to create a support case, please let us know here and we will be happy to help create one from our end.

#6

Hello @tlugo,

I have a similar issue with PHP 8.1 and the New Relic PHP agent. PHP 8.1 is updated to it’s latest version (8.1.6) and the New Relic agent is also updated to the latest version (9.21.0.311) . Both to no avail we still the see majority of the transactions are being labeled as unknown (97%).

Is there anything else we could try to solve this issue?

#7

Hi @robin3

Thanks for reaching out.

I will likely need to loop in an engineer here however can you provide a permalink to where you are noting this.

Only New Relics will have access to this link, this will grant us more clarity into why its occurring.

Looking forward to hearing from you!

#8

Are there any updates to this issue ?

We started having the same problem after upgrading to PHP 8.1.5
New Relic reported all transactions as “unknown”
Our agent has been updated to 9.21.0.311

#9

Hi @tlugo

You mean you want a URL to the dashboard which shows all the transactions for the account? If so then I think this should work for you: https://onenr.io/0vwB1v0BXQp

#10

I’m experiencing the same issues after installing OPcache. All Laravel transactions are showing up as /unknown, and the errors make it to the laravel.log file but are not being reported in NewRelic. I have some non-Laravel PHP files on the same server that work just fine (transactions are properly named and errors are properly reported).

My NewRelic instrumentation is essentially useless in this state.

PHP (8.1.6-fpm)
Zend OPcache (8.1.6)
Upgrading the NewRelic PHP agent from 9.20.0.310 to 9.21.0.311 did not help.

Disabling OPcache does resolve the problem but is not a viable option.

#11

Hey @brett12, welcome to the community!

@andrew.bergamasco, welcome back, it is good to see you again!

I wanted to let you, as well as @robin3, that the team is still reviewing this and will provide an update shortly. We really appreciate your patience as we provide assistance with this issue. Please let us know if there are any updates on your end and don’t hesitate to ask any questions you may have.

#12

Hi, we are experiencing similar issues.

PHP (8.1.6-fpm)
Symfony (4.4)
NewRelic PHP agent upgraded from 9.9.0.260 to 9.21.0.311

As part of our upgrade from PHP 7.3 to 8.1 we’ve encountered the following issues.

Firstly when we upgraded from PHP 7.3 to 8.1, resulted in Newrelic APM not working. To recover this, we upgraded to the latest version of Newrelic (9.21.0.311), which brought Newrelic back.

After the Newrelic agent upgrade, we’ve now seen performance issues. Deep in the Symfony start up process we receive the following:

message:NOTICE: PHP message: PHP Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/vendor/symfony/dependency-injection/Definition.php

When we see this error, the pod/s do not recover and many different approaches do not result in the pod recovering. We tested many different things to get the pod/s to recover, but it appears that only removing the Newrelic agent is successful. See our steps below:

  • Cycled the deployment - does not fix the issue
  • Deleted the affected pods - does not fix the issue
  • Deployed new version of PHP 8.1.6 from 8.1.5 (processes update) - does not fix the issue
  • Remove Newrelic agent from the Node - This resolves the issues
  • Rolled back Newrelic agent from 9.21.0.311 to 9.9.0.260 - This also resolves the issue

We are now in the process of rolling back to agent 9.9.0.260 in production, due to the issues mentioned above. We also don’t know how to replicate the issue and it’s very intermittent. Running the upgrade in production for over a month has only resulted in several occurrences.

What we see is auto scaling of pods in Kubenetes with a high increase in CPU on certain agents. Once we receive the timeout and the increase of CPU occurs on a given agent, the pod running on that agent does not recover unless we remove Newrelic.

I’m happy to provide more information or any screenshots of logging that may help you investigate further?

Thanks

1 Like
#13

Hey there @leigh.mills,

Welcome to the community!

I am looping in a support engineer to help out here as it is a bit out of my scope. We appreciate your patience as we work with you and our team will respond here shortly.

#14

Hello @leigh.mills ,

Thanks for your patience. I ran this by our engineers and they noted that the message you are seeing is a PHP (not symfony) error:

It’s possible you are seeing it with the new agent due to the increased overhead of Distributed Tracing now being on by default.

To support distributed tracing, the PHP agent needs to keep a significant amount of function call data at runtime so that spans can be sampled as accurately as possible. For longer background transactions or transactions that perform a series of repetitive processes, such as a message queue consumer or a loop with multiple external calls, the increased overhead may become noticeable.

You might try either lowering the max_samples_stored or disabling Distributed Tracing to see if either of those helps.

I hope this helps!

#15

Hello,

Is there any update regarding this?

#16

Hi there @robin3 - It does not look like anyone has feedback about @ntierney’s advice. Have you had a chance to try his suggestions? Or were you looking for an update on something else?

#17

Hi @hross,

The framework that is used is not a Symfony but Laravel in our case so we don’t get those error messages which makes me think that the suggestion won’t change anything. We haven’t found any errors in the logs on the servers so we only see a lot of /unknown messages in New Relic since the upgrade to PHP 8.1.

#18

Hey @robin3,

Thank you for the clarification. We are going to loop in an expert from our PHP team to help further. We appreciate your patience while we provide support and will get a response out to you shortly.

#19

Hello @robin3,

This sounds like a known issue our team is aware of, although I don’t have a timeline of when a fix will go out.

There are some workarounds available in the Github issue that might help!

#20

Morning,

Thank you for the reply and the information.

We had distributing tracing disabled previously and still do, so the only other possible thing we can try is lowering the samples stored. Is lowering the samples stored still relevant with distributed tracing disabled?

Please let us know if you have any further suggestions or information as to what’s causing this.

Thanks