Updated JVM to use G1GC but not seeing any metrics in APM - JVM

Hi New Relic,

Recently we’ve updated one of our Java apps to use G1GC garbage collector. We’ve updated it on our test environments with no issue (I can monitor JVM metrics fine on APM), but when we’ve made the change on 1 of our production servers to smoke test, we’re not seeing anything in JVM to indicate GC metrics / heap size / etc. I’ve verified the JVM settings are running on the server fine.

Today there was an issue with APM UI chart irregularities but has since been resolved. I am curious if this may be due to the heap size being large (50G heap) and the amount of metrics or if there might be an issue due to the same application having two garbage collectors on different instances within the same APM? I wouldn’t think that’s the case seeing as the existing CMS GC shows fine on all the other instances.

Not seeing anything in the newrelic-java agent logs

Nov 18, 2019 20:47:28 +0000 [8954 6] com.newrelic.agent.Agent INFO: New Relic Agent has shutdown
Nov 18, 2019 20:49:02 +0000 [20101 1] com.newrelic INFO: Writing to New Relic log file: /var/log/newrelic-java/newrelic_agent.log
Nov 18, 2019 20:49:02 +0000 [20101 1] com.newrelic INFO: JRE vendor Amazon.com Inc. version 1.8.0_212
Nov 18, 2019 20:49:02 +0000 [20101 1] com.newrelic INFO: JVM vendor Amazon.com Inc. OpenJDK 64-Bit Server VM version 25.212-b04
Nov 18, 2019 20:49:02 +0000 [20101 1] com.newrelic INFO: OS Linux version 4.4.0-1084-aws arch amd64
Nov 18, 2019 20:49:02 +0000 [20101 1] com.newrelic INFO: Agent Host: [MYSERVERNAME] IP: myipaddress
Nov 18, 2019 20:49:02 +0000 [20101 1] com.newrelic INFO: New Relic Agent v4.11.0 is initializing...
Nov 18, 2019 20:49:05 +0000 [20101 1] com.newrelic.agent.RPMServiceManagerImpl INFO: Configured to connect to New Relic at collector.newrelic.com:443
Nov 18, 2019 20:49:05 +0000 [20101 1] com.newrelic INFO: Setting audit_mode to false
Nov 18, 2019 20:49:05 +0000 [20101 1] com.newrelic INFO: Setting protocol to "https"
Nov 18, 2019 20:49:06 +0000 [20101 1] com.newrelic.agent.config.ConfigServiceImpl INFO: Configuration file is /opt/newrelic-java/newrelic/./newrelic.yml
Nov 18, 2019 20:49:06 +0000 [20101 1] com.newrelic INFO: New Relic Agent v4.11.0 has started
Nov 18, 2019 20:49:06 +0000 [20101 1] com.newrelic INFO: Agent class loader: com.newrelic.bootstrap.BootstrapAgent$JVMAgentClassLoader@21b8d17c
Nov 18, 2019 20:49:06 +0000 [20101 1] com.newrelic INFO: Premain startup complete in 3,858ms
Nov 18, 2019 20:49:11 +0000 [20101 29] com.newrelic INFO: Host name is [MYSERVERNAME], display host [MYSERVERNAME] for application [MYSERVERNAME]
Nov 18, 2019 20:49:11 +0000 [20101 29] com.newrelic INFO: Collector redirection to collector-001.newrelic.com:443
Nov 18, 2019 20:49:11 +0000 [20101 29] com.newrelic INFO: Max payload size is 1,000,000 bytes
Nov 18, 2019 20:49:11 +0000 [20101 29] com.newrelic INFO: Agent run id: BVEBYZWWO8U-AAS9FAPl-tZd0wQ0kkjlsaflkjsadfkljadfsjdklfasjklasfjkladsflkjafsd
Nov 18, 2019 20:49:11 +0000 [20101 29] com.newrelic INFO: Agent 20101@[MYSERVERNAME]/[MYSERVERNAME] connected to collector.newrelic.com:443
Nov 18, 2019 20:49:11 +0000 [20101 29] com.newrelic INFO: Reporting to: https://rpm.newrelic.com/accounts/310548/applications/48860313
Nov 18, 2019 20:49:11 +0000 [20101 29] com.newrelic INFO: Using default collector host: collector.newrelic.com
Nov 18, 2019 20:49:11 +0000 [20101 29] com.newrelic INFO: Using default collector host: collector.newrelic.com
Nov 18, 2019 20:49:11 +0000 [20101 29] com.newrelic INFO: Real user monitoring is enabled for application "[MYAPPNAME]"
Nov 18, 2019 20:51:47 +0000 [20101 220] com.newrelic INFO:[MYSERVERNAME] is reconnecting
Nov 18, 2019 20:51:47 +0000 [20101 29] com.newrelic INFO: Host name is [MYSERVERNAME], display host [MYSERVERNAME] for application [MYSERVERNAME]
Nov 18, 2019 20:51:48 +0000 [20101 29] com.newrelic INFO: Collector redirection to collector-001.newrelic.com:443
Nov 18, 2019 20:51:48 +0000 [20101 29] com.newrelic INFO: Max payload size is 1,000,000 bytes
Nov 18, 2019 20:51:48 +0000 [20101 29] com.newrelic INFO: Agent run id: BREXmKmCjdTbAAS9FAfgR7pd0wSsAAIBAAAnIQEAAE6FAgQC6YyZAwAGNC4xMS4wAAxQcm9kLUlBTS0wMWQACElBTS1Qcm9k
Nov 18, 2019 20:51:48 +0000 [20101 29] com.newrelic INFO: Agent 20101@[MYSERVERNAME][MYSERVERNAME] connected to collector.newrelic.com:443
Nov 18, 2019 20:51:48 +0000 [20101 29] com.newrelic INFO: Reporting to: https://rpm.newrelic.com/accounts/310548/applications/48860313
Nov 18, 2019 20:51:48 +0000 [20101 29] com.newrelic INFO: Using default collector host: collector.newrelic.com
Nov 18, 2019 20:51:48 +0000 [20101 29] com.newrelic INFO: Using default collector host: collector.newrelic.com
Nov 18, 2019 20:51:48 +0000 [20101 29] com.newrelic INFO: Real user monitoring is enabled for application "MYAPPNAME"
Nov 18, 2019 20:54:18 +0000 [20101 330] com.newrelic WARN: Dropped 201 span events out of 1,201.
Nov 18, 2019 20:55:18 +0000 [20101 330] com.newrelic WARN: Dropped 895 span events out of 1,895.
Nov 18, 2019 20:55:19 +0000 [20101 330] com.newrelic WARN: Dropped 191 error events out of 291.
Nov 18, 2019 20:56:19 +0000 [20101 330] com.newrelic WARN: Dropped 23 error events out of 123.
Nov 18, 2019 20:59:33 +0000 [20101 657] com.newrelic WARN: Custom event with invalid attributes key or value of null was reported for a transaction but ignored. Each key should be a String and each value should be a String, Number, or B
oolean.
Nov 18, 2019 21:00:10 +0000 [20101 575] com.newrelic WARN: Custom event with invalid attributes key or value of null was reported for a transaction but ignored. Each key should be a String and each value should be a String, Number, or B
oolean.
Nov 18, 2019 21:00:10 +0000 [20101 575] com.newrelic WARN: Custom event with invalid attributes key or value of null was reported for a transaction but ignored. Each key should be a String and each value should be a String, Number, or B
oolean.
Nov 18, 2019 21:00:32 +0000 [20101 656] com.newrelic WARN: Custom event with invalid attributes key or value of null was reported for a transaction but ignored. Each key should be a String and each value should be a String, Number, or B
oolean.
Nov 18, 2019 21:00:32 +0000 [20101 656] com.newrelic WARN: Custom event with invalid attributes key or value of null was reported for a transaction but ignored. Each key should be a String and each value should be a String, Number, or B
oolean.
Nov 18, 2019 21:15:55 +0000 [20101 616] com.newrelic WARN: Custom event with invalid attributes key or value of null was reported for a transaction but ignored. Each key should be a String and each value should be a String, Number, or B
oolean.
Nov 18, 2019 21:16:14 +0000 [20101 665] com.newrelic WARN: Custom event with invalid attributes key or value of null was reported for a transaction but ignored. Each key should be a String and each value should be a String, Number, or B
oolean.
Nov 18, 2019 21:16:14 +0000 [20101 665] com.newrelic WARN: Custom event with invalid attributes key or value of null was reported for a transaction but ignored. Each key should be a String and each value should be a String, Number, or B
oolean.
@

While on our staging environment the same JVM metrics are showing fine.

1 Like

The issue was due to our APM metrics being whitelisted due to a metric group issue.

Thanks for following up with the cause of this issue @ian.arsenault

:crossed_fingers: Hope you can get that MGI resolved.

I am facing the same issue now. I am trying to understand this solution better.
I upgraded the NR agent to the latest version but the issue still persists.

Can New Relic team please help?

Hi @shankar.pachari,

I posted a reply to your post here: Java Garbage Collection Metrics Not Showing Up After Updating Gc Strategy To Xx Useg1Gc