[PHP] Guzzle errors in PHP Agent 9.0+

After we upgraded to PHP Agent 9.0+ and PHP 7.3+ we began getting random errors in Guzzle client.

Example:
“Unsupported operand types”
Trace:
#0 /vendor/guzzlehttp/guzzle/src/Client.php(111): GuzzleHttp\Client->prepareDefaults(Array)
#1 /vendor/guzzlehttp/guzzle/src/Client.php(131): GuzzleHttp\Client->requestAsync(‘shouldReport’, Array, Array)
#2 /vendor/guzzlehttp/guzzle/src/Client.php(89): GuzzleHttp\Client->request(‘shouldReport’, Array, Array)
#3 /vendor/guzzlehttp/guzzle/src/Client.php(62): GuzzleHttp\Client->__call(‘shouldReport’, Array)
#4 /app/Services/CallbackService.php(291): GuzzleHttp\Client->__construct(Array)

“InvalidArgumentException: Magic request methods require a URI and optional options array in /vendor/guzzlehttp/guzzle/src/Client.php:81”
Trace:
#0 /app/Services/Streamer/StreamerManager.php(31): GuzzleHttp\Client->__call(‘getConnection’, Array)

As if in the Guzzle constructor methods are called that should never be called in the context.
The issue only shows in long-running transactions, in particular, Laravels queue worker daemons. After we restart the daemon processes, the issue disappears for some time.

After downgrading to 8.7 PHP agent the issue disappeared.

Hi @andrei9,

Thanks for your post here in the community! Our team is wondering which version of Guzzle and Laravel you are using. That would help us in narrowing down code paths and trying to come up with a reproduction. A permalink to the application would also be helpful and would only be visible to New Relic support. Thanks!

Hi

Laravel 5.7, but bug was also present in previous versions
Guzzle version 6.3.0

https://rpm.newrelic.com/accounts/1154658/applications/170350466

@andrei9 Thank you for providing that information, I’ve passed it along to our engineering team, and they are looking into this.

The issue you describe sounds very similar to a known snag some users have encountered on the latest agents: Potential memory exhaustion for long running transactions with PHP agent 9.x

Our engineers are working hard on a fix for the issue I linked, and I am curious whether your application was showing similarly high memory usage, since it’s specifically happening with these long-running background processes.

Yes, the memory leak issue was the second one we had with 9.0+ version.

@andrei9 that is interesting then, it seems that there is a correlation between these odd errors and the system running out of memory.

Our engineering team is coming out with a fix for the memory issue in the very near future, and hopefully that will solve this as well. Keep an eye on our PHP Agent Release Notes for the new version; for the time being, the recommendation is to stay on the 8.7 version for any hosts running those long transactions.

We’ve been seeing these exact same errors when running v9.9.0.260 of the PHP agent (as well as some earlier v9 releases—we’ve been upgrading pretty regularly) using the install script, in an Alpine Linux Docker environment, with the agent and daemon in the same container. They’ve also been occurring in a long-running Laravel queue worker. Our memory usage always hovers around 15%. We’ve had a few deployments where the error doesn’t manifest itself, but when it does it’s immediately on deployment, so it doesn’t sound related to a memory leak. The only file that seems to show this weird behavior is the Guzzle Client class, and it doesn’t matter if the instance is being built via the Laravel service container and the PHP reflection API or if we instantiate it directly.

We’ve since rolled back to v8.7.0.242 of the PHP agent and the errors disappeared.

Here’s another stack trace that seems to be the same thing:

{
    "class": "InvalidArgumentException",
    "message": "Magic request methods require a URI and optional options array",
    "code": 0,
    "file": "/home/www/app/vendor/guzzlehttp/guzzle/src/Client.php:81",
    "trace": [
        "/home/www/app/app/Services/SomeClass.php:29",
        "{\"function\":\"__construct\",\"class\":\"App\\\\Services\\\\SomeClass\",\"type\":\"->\",\"args\":[]}",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:800",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:646",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:601",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:734",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:900",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:828",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:795",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:646",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:601",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:734",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:900",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:828",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:795",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:646",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:601",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:734",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:160",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:116",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:29",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:87",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:31",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:564",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php:94",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:116",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:104",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php:98",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php:49",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php:83",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:326",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:276",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:118",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:101",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:85",
        "{\"function\":\"handle\",\"class\":\"Illuminate\\\\Queue\\\\Console\\\\WorkCommand\",\"type\":\"->\",\"args\":[]}",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:29",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:87",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:31",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:564",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Console/Command.php:179",
        "/home/www/app/vendor/symfony/console/Command/Command.php:255",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Console/Command.php:166",
        "/home/www/app/vendor/symfony/console/Application.php:908",
        "/home/www/app/vendor/symfony/console/Application.php:269",
        "/home/www/app/vendor/symfony/console/Application.php:145",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Console/Application.php:89",
        "/home/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:122",
        "/home/www/app/artisan:35"
    ]
}

These stack traces always indicate strange execution paths that shouldn’t be possible, where Client::__call() is invoked from Client::__construct().

Also, since downgrading to v8 of the PHP agent a job that we run every five minutes has begun to report much more consistently. Before the downgrade, only about 50-60% of the transactions would be reported in hit-or-miss fashion.

@jwisse thank you for that info, I’m going to be passing this along to our Agent team for investigation. We have been digging into how the Agent behaves with long-running CLI processes for a bit now, and this will be helpful.

Hello @scandela, i would like to follow up on this. Having the same issue as @jwisse

{
    "class": "InvalidArgumentException",
    "message": "Magic request methods require a URI and optional options array",
    "code": 0,
    "file": "/home/www/app/vendor/guzzlehttp/guzzle/src/Client.php:81",
    "trace": [

@winn.minn thank you for sharing this. We are going to try and reproduce this; there was a rework of curl_multi_exec and some Guzzle instrumentation in recent versions of the Agent, so we’re investigating and will let you know what we find!

Hi @scandela, we’re also experiencing this issue (we’re on the latest version 9.13.0.270) - if there is any additional information that we can provide let us know.

We’re using Guzzle 6.5.5 and Laravel 7.22.4 if that’s of any assistance.

Could you show us the current situation of this issue?

Hi @iwamoto, @nick15, @winn.minn, and @jwisse - I do not have an update for you yet, unfortunately. Engineering has this and is working to reproduce and diagnose. I can’t tell you when we will have an answer, but please know that it is being looked at. We’ll give an update as soon as we can.

A post was split to a new topic: When to call GetConnection

Could you show us the current situation of this issue?

This is happening to us too. Are there any updates?

Running into the same issue here too. After installing NewRelic last night my Laravel based PHP 7.x app has been throwing errors for all Guzzle calls. Had to disable it to get everything back to ‘normal.’

Hi all,

We are experiencing the same problems on a Laravel 7 application.
Many of the queued jobs fail because of this problem. When pushing the failed jobs individually for a retry, they work without a problem (without restarting the background workers).

For now, we’ve disabled the New Relic agents; but by reading the others posts I will ask our hosting provider to install 8.x version.

Unfortunately is downgrading to 8.x agent impossible. This version of the agent does not support PHP 7.4; so basically we’re stuck. For now we have no other option than disabling the New Relic agent.