Potential memory exhaustion for long running transactions with PHP agent 9.x

PHP agent 9.x uses more memory than previous versions, as it is less aggressive about freeing the memory used to track function calls and segments during transactions: in the normal case, memory is only freed at the end of each transaction.

We have been working through reported cases of memory exhaustion with PHP agent 9.x and believe that we have fully characterized this issue. This tends to manifest for users with long running transactions, such as background jobs to process message queues, transform or report on data, or send e-mails.

We apologize for the inconvenience that this issue has caused, and are actively working to fix it. In the short term, we have several options available to mitigate this, described further on in this posting.

Why did we do this?

Version 9.0 of the PHP agent added significantly improved support for distributed tracing. Versions 8.5 to 8.7 (inclusive) of the PHP agent included distributed tracing support, but required a set of configuration settings that made general PHP application monitoring significantly more coarse-grained.

To support the improved distributed tracing support and future agent features, PHP agent 9.0 onwards need to keep significantly more function call data at runtime so that spans can be sampled as accurately as possible. For web transactions and shorter background transactions, this is generally unnoticeable, both in terms of CPU and memory overhead. However, for longer background transactions, the increased memory overhead may become noticeable, even to the point of resulting in an out of memory condition.

What are we doing about it?

We are working on adding more configuration settings to control memory usage by the PHP agent. Broadly speaking, there’s a trade-off between CPU overhead and memory overhead required for the PHP agent to gather the data needed to support deeper visibility into your PHP services and wide visibility in your cross-service transactions, and we need to provide more controls for users to choose where on that spectrum they want to sit.

We are also exploring whether we need to modify the default behavior for background transactions so that memory is freed more frequently and aggressively for longer running transactions.

We are currently working on adding these configuration settings and exploring the change to the default behavior. We forecast several more weeks before we can deliver this in totality. In the meantime, we will release PHP agent support for PHP 7.4, and we anticipate that we will have a phased delivery of the new functionality needed to provide memory usage controls.

In the interim, we have some suggested workarounds, which we’ve listed below.

Workarounds available today

1. Start and stop transactions manually

Pros

  • Allows all PHP agent features to remain enabled.
  • Provides more fine-grained data on how the process is operating.

Cons

  • Requires code changes.

If the affected transaction is one that performs a series of repetitive processes, such as a message queue consumer, you may wish to manually instrument each iteration as a separate transaction. By doing this, the memory used will be freed after each transaction.

To implement this, you would ignore the initial automatic transaction with newrelic_end_transaction(true), then use newrelic_start_transaction() and newrelic_end_transaction() to instrument each transaction in turn.

A rough approximation of how this might look in practice would be:

<?php

newrelic_end_transaction(true);

// ...

while ($message = $queue->next()) {

   newrelic_start_transaction('Application Name');
   newrelic_name_transaction($message->getName());

    // ...

   newrelic_end_transaction(false);
}

2. Reduce transaction tracer detail

Pros

  • This is easy and might be sufficient for some cases. (Note: this change will occur automatically when you turn on distributed tracing for the PHP agent 8.x releases that support distributed tracing)
  • Allows you to keep up with the latest bug fixes and features in the PHP agent.
  • Full distributed tracing support is available.

Cons

  • Traces will no longer contain PHP function calls.
  • Transactions that make hundreds of thousands of datastore or external calls may still be affected by memory issues.

If you need PHP 7.4 or any of the functionality added in the PHP agent 9.x releases, or distributed tracing support, and can get by with traces only including information on datastore and external calls, then you can reduce the level of detail that the PHP agent captures by changing this configuration setting:

newrelic.transaction_tracer.detail = 0

3. Downgrade to PHP agent 8.7

Pros

  • Uses a stable version of the PHP agent with most of the features of the current version.
  • Under no circumstances will more than 2,000 segments be created, which limits memory usage.

Cons

  • We will not be shipping PHP 7.4 support for the version 8 PHP agent.
  • If distributed tracing is enabled on PHP agent 8.7, transaction tracer detail is automatically reduced (as per the “reduce transaction tracer detail” option).

If you don’t need distributed tracing support with detailed transaction traces, and don’t intend to use PHP 7.4 in the next few months, this is likely the simplest and fastest solution.

To downgrade, you can either install from the tarballs at https://download.newrelic.com/php_agent/archive/8.7.0.242/, or downgrade to version 8.7.0.242 in your package manager and pin that version.

4. Cap the number of function segments that are created

Pros

  • Same as those under “reduce transaction tracer detail”, plus PHP function calls are instrumented.
  • This is easy and might be sufficient for some specific cases.

Cons

  • Memory usage is higher than the “reduce transaction tracer detail” option to cover the number of PHP functions that are traced.
  • Any PHP functions after the number of function calls that are configured will be ignored.
  • Transactions that make hundreds of thousands of datastore or external calls may still be affected by memory issues.

This is similar to the “reduce transaction tracer detail” option, but also allows for a limited number of PHP function calls to be traced, at the cost of an increase in memory usage commensurate with the number of functions that are captured. (As a rough rule of thumb, each segment requires about 320-400 bytes of the heap.)

To capture the first 5,000 function calls, you would add this configuration setting:

newrelic.transaction_tracer.max_segments = 5000

Where else can I find information about this?

We have updated our release notes to capture this as a “known issue” and included our proposed workarounds. The content here in this Explorer’s Hub post is more comprehensive than what is currently contained in the release notes.

The release notes will also indicate when this issue is fixed.

Please do continue to report this issue when you see it, and any information on your use case that you can share. This helps us to confirm that we have fully characterized this issue.

Sincerely,

Jodee Varney
Product Manager, New Relic PHP agent

3 Likes