Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Agent version migration causes memory exhaustion

scala
play
javaagent

#1

Hello!
We migrate the Java agent from version 3.30.0 to version 4.5.0.
After some days the application is running we notice that memory (Old heap) allways grows and the GC do not clean all the objects.
After doing a memory dump and open it with Visual VM we see that between the most consumption objecs, many were NewRelic instances

After downgrade the version back to 3.30.0, the memory problems disapears completely

Here you can see the difference between versions (the version downgrade was applied at Oct 19)

A detail, betweens agent version we do not change the conf file. This file contains:

common: &default_settings
license_key: ‘XXXXXXXXXXXX’
enable_auto_app_naming: false
enable_auto_transaction_naming: true
enable_custom_tracing: true
app_name: XXXXX
log_level: info
ssl: false
capture_params: false
transaction_tracer:
enabled: true
transaction_threshold: apdex_f
record_sql: obfuscated
log_sql: false
stack_trace_threshold: 0.5
explain_enabled: true
explain_threshold: 0.5
error_collector:
enabled: true
ignore_errors: java.nio.channels.ClosedChannelException
ignore_status_codes: 404
thread_profiler:
enabled: true
browser_monitoring:
auto_instrument: true
enabled: true
prod:
<<: *default_settings
beta:
<<: *default_settings
app_name: XXXX (BETA)
license_key: ‘XXXXXXXX’

Any idea the reason of the memory problem?
Our application is using the 3.30.0 version for several days and the memory is stable.

Hope you can help us
Thanks!


#2

Hi @fernando.giannetti,

To further investigate this issue. Kindly send us a permalink to the affected application? To create a permalink to any page within the New Relic APM UI, scroll to the bottom and click ‘Permalink’ - all the way on the right next to ‘Kiosk Mode. I’d like to get the permalink to look at some metrics from your app and to determine what underlining components are being utilized.

Regards,


#3

Hi @bidahor!
Sorry the delay of the response. I never get the email :slight_smile:
This is the permalink that compares both agent versions (the image of the original post). Hope be helpfull.
https://rpm.newrelic.com/accounts/1211719/applications/18557486/instances?tw[end]=1540223128&tw[start]=1539618328

Thanks!


#4

Hi @fernando.giannetti,

Thank you for sharing the permalink. While investigating this memory exhaustion issue, I noticed you have an older agent(v3.48.0) with Play 2.5.18 running on the app. First, I would suggest an agent upgrade to v4.8.0 on this app . Lets give that a try and see if that improves things on your end.

Best,


#5

Thanks for your answer @bidahor!
Im using this older agent (v3.48.0) after rollback the v4.5.0 because the memory exhaustion.
I will try the 4.8.0 like you say.
Will see what happen :slight_smile:

Thanks!


#6

Hi @fernando.giannetti - I’m glad you got a good suggestion from @bidahor. Let us know when you find out if that worked or not so that other will also know how to resolve this issue.


#7

@bidahor
Using the 4.9.0 version, the issue is still there.
https://rpm.newrelic.com/accounts/1211720/applications/31439279/instances?tw[end]=1547037946&tw[start]=1546994746
(ignore the red zone is a testing load. The Old heap still grows all the time)

Thanks!


#8

Hi @fernando.giannetti,

Thank you for the update. I’ll need some other information(e.g logs and heap dumps) to further investigate this issue. I will open an internal ticket with Support for us to continue investigating this memory exhaustion issue.

Thanks!


#9

Hello!
This issue is fixed using the 4.10.0 version of the agent.
Thank you very much for all your help!


#10

@bidahor
Hello again :slight_smile:
After more than a week the memory problem is back.
You can see it in the permalink https://rpm.newrelic.com/accounts/1211719/applications/18557486/instances?tw[end]=1550237460&tw[start]=1549632660#id=176520785

Thanks!


#11

Hi @fernando.giannetti,

I have opened a follow-up to your internal ticket to continue troubleshooting.

Thanks!