I am trying to reduce the daily amount of data ingested for tracing.
Through “Manage your data” => “Data ingestion” I can see that I am currently using almost 3GB per day for “Tracing”.
When I go to “Services - APM” => “Transactions”, under “Transaction traces”, I see (as expected):
“No transaction traces above threshold in the last 3 hours.
If you were expecting traces, no transactions took longer than 2.0 seconds (4 * apdex_t) or there is an error in the agent configuration.”
This is as expected, apdex_t is set to 0.5s in “Services - APM” => “Application”.
Relevant settings in the newrelic.ini (confirmed in “Services - APM” => “Environment” => “Agent initialization”):
;newrelic.transaction_tracer.enabled = true
;newrelic.transaction_tracer.threshold = “apdex_f”
;newrelic.distributed_tracing_enabled = true
(All those settings are commented, but the settings are the defaults).
So “Services - APM” => “Transactions”, under “Transaction traces” listens to these settings and does not store/show anything below 4x apdex_t (2 secs in this case).
However, when I go to “Services - APM” => “Distributed tracing”, I see over 16 thousand traces for that same 3 hour period, including traces for transactions with a duration of 20ms, 100ms, 200ms, etc (all not even close to apdex_f).
I suspect that these traces/span info is what is using almost 3GB per day.
When I look at the docs for “Distributed tracing for the PHP agent” here: Distributed tracing for the PHP agent | New Relic Documentation
Under step 4, regarding the newrelic.transaction_tracer.threshold it reads: “If you want to make all transactions eligible for a distributed trace, set this value to 0 seconds.”
However, this is set to the default value of apdex_f, yet this seems to be ignored for distributed tracing, since I still get distributed traces for transactions, since I see 16K traces in the last 3 hours all nowhere near apdex_f in duration.
Is this expected/intended behavior or a bug?
If it’s intended, then maybe step 4 on Distributed tracing for the PHP agent | New Relic Documentation should be removed or replaced with a notice stating that for distributed tracing the newrelic.transaction_tracer.threshold setting is ignored, because currently it obviously implies that that setting influences distributed tracing eligibility as well.
I could of course disable distributed tracing completely, but it would still be nice to be able to see traces for slow transactions (above the threshold).
I am using PHP Agent version: 126.96.36.1993