Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Feature Idea: Support for Play 1.5


#1

Hi, I’m using Java Agent v3.42 with a Play1 (v1.5) app running on Java8 and I’ve started to get these NewRelic exceptions after updating to v3.42. Any ideas? I’ve added the start up log further down. Note also the “ERROR: Inconsistent state! tracer != last tracer” in the log. best, Chris

2017-09-18T00:40:17.710310+00:00 heroku[router]: at=info method=POST path="/events/30125/ticketPurchases" host=app.ptasocial.com request_id=e5873fc9-f561-485b-b8da-b5d69865cc98 fwd="2604:2000:7009:f100:e400:c997:6c09:a615,108.162.219.81" dyno=web.1 connect=0ms service=158ms status=200 bytes=4815 protocol=https
2017-09-18T00:40:21.485105+00:00 app[web.1]: Sep 18, 2017 00:40:21 +0000 [43 24] com.newrelic.agent.TransactionService WARN: Exception processing async update queue.
2017-09-18T00:40:21.485117+00:00 app[web.1]: java.lang.NullPointerException: null
2017-09-18T00:40:21.485119+00:00 app[web.1]: 	at com.newrelic.agent.Transaction.checkExpireTracedActivities(Transaction.java:2262) ~[newrelic.jar:3.42.0]
2017-09-18T00:40:21.485123+00:00 app[web.1]: 	at com.newrelic.agent.Transaction.cleanUp(Transaction.java:1096) ~[newrelic.jar:3.42.0]
2017-09-18T00:40:21.485123+00:00 app[web.1]: 	at com.newrelic.agent.TransactionService.processQueue(TransactionService.java:81) ~[newrelic.jar:3.42.0]
2017-09-18T00:40:21.485124+00:00 app[web.1]: 	at com.newrelic.agent.TransactionService$1.run(TransactionService.java:69) [newrelic.jar:3.42.0]
2017-09-18T00:40:21.485125+00:00 app[web.1]: 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144-cedar14]
2017-09-18T00:40:21.485126+00:00 app[web.1]: 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_144-cedar14]
2017-09-18T00:40:21.485127+00:00 app[web.1]: 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_144-cedar14]
2017-09-18T00:40:21.485128+00:00 app[web.1]: 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_144-cedar14]
2017-09-18T00:40:21.485128+00:00 app[web.1]: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144-cedar14]
2017-09-18T00:40:21.485129+00:00 app[web.1]: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144-cedar14]
2017-09-18T00:40:21.485130+00:00 app[web.1]: 	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144-cedar14]
2017-09-18T00:40:51.486150+00:00 app[web.1]: Sep 18, 2017 00:40:51 +0000 [43 24] com.newrelic.agent.TransactionService WARN: Exception processing async update queue.
2017-09-18T00:40:51.486280+00:00 app[web.1]: java.lang.NullPointerException: null
2017-09-18T00:40:51.486282+00:00 app[web.1]: 	at com.newrelic.agent.Transaction.checkExpireTracedActivities(Transaction.java:2262) ~[newrelic.jar:3.42.0]
2017-09-18T00:40:51.486284+00:00 app[web.1]: 	at com.newrelic.agent.Transaction.cleanUp(Transaction.java:1096) ~[newrelic.jar:3.42.0]
2017-09-18T00:40:51.486285+00:00 app[web.1]: 	at com.newrelic.agent.TransactionService.processQueue(TransactionService.java:81) ~[newrelic.jar:3.42.0]
2017-09-18T00:40:51.486286+00:00 app[web.1]: 	at com.newrelic.agent.TransactionService$1.run(TransactionService.java:69) [newrelic.jar:3.42.0]
2017-09-18T00:40:51.486287+00:00 app[web.1]: 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144-cedar14]
2017-09-18T00:40:51.486288+00:00 app[web.1]: 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_144-cedar14]
2017-09-18T00:40:51.486289+00:00 app[web.1]: 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_144-cedar14]
2017-09-18T00:40:51.486290+00:00 app[web.1]: 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_144-cedar14]
2017-09-18T00:40:51.486291+00:00 app[web.1]: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144-cedar14]
2017-09-18T00:40:51.486292+00:00 app[web.1]: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144-cedar14]
2017-09-18T00:40:51.486293+00:00 app[web.1]: 	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144-cedar14]
2017-09-18T00:40:56.864393+00:00 heroku[router]: at=info method=PATCH path="/eventTicketPurchases/949" host=app.ptasocial.com request_id=2957796b-63b4-4b3b-987f-e5c6dad4f0e3 fwd="2604:2000:7009:f100:e400:c997:6c09:a615,108.162.219.79" dyno=web.1 connect=0ms service=1035ms status=500 bytes=803 protocol=https
2017-09-18T00:40:56.862274+00:00 app[web.1]: Sep 18, 2017 00:40:56 +0000 [43 81] com.newrelic ERROR: Inconsistent state!  tracer != last tracer for com.newrelic.agent.TransactionActivity@0 (com.newrelic.agent.instrumentation.pointcuts.play.PlayDispatcherPointCut$1@5f2922bb != com.newrelic.agent.tracers.DefaultTracer@7afd0d47)
2017-09-18T00:41:21.486945+00:00 app[web.1]: Sep 18, 2017 00:41:21 +0000 [43 24] com.newrelic.agent.TransactionService WARN: Exception processing async update queue.
2017-09-18T00:41:21.486955+00:00 app[web.1]: java.lang.NullPointerException: null
2017-09-18T00:41:21.486956+00:00 app[web.1]: 	at com.newrelic.agent.Transaction.checkExpireTracedActivities(Transaction.java:2262) ~[newrelic.jar:3.42.0]
2017-09-18T00:41:21.486957+00:00 app[web.1]: 	at com.newrelic.agent.Transaction.cleanUp(Transaction.java:1096) ~[newrelic.jar:3.42.0]
2017-09-18T00:41:21.486957+00:00 app[web.1]: 	at com.newrelic.agent.TransactionService.processQueue(TransactionService.java:81) ~[newrelic.jar:3.42.0]
2017-09-18T00:41:21.486958+00:00 app[web.1]: 	at com.newrelic.agent.TransactionService$1.run(TransactionService.java:69) [newrelic.jar:3.42.0]
2017-09-18T00:41:21.486958+00:00 app[web.1]: 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144-cedar14]
2017-09-18T00:41:21.486959+00:00 app[web.1]: 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_144-cedar14]
2017-09-18T00:41:21.486960+00:00 app[web.1]: 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_144-cedar14]
2017-09-18T00:41:21.486961+00:00 app[web.1]: 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_144-cedar14]
2017-09-18T00:41:21.486962+00:00 app[web.1]: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144-cedar14]
2017-09-18T00:41:21.486962+00:00 app[web.1]: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144-cedar14]
2017-09-18T00:41:21.486963+00:00 app[web.1]: 	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144-cedar14]
2017-09-18T00:41:51.488077+00:00 app[web.1]: Sep 18, 2017 00:41:51 +0000 [43 24] com.newrelic.agent.TransactionService WARN: Exception processing async update queue.
2017-09-18T00:41:51.488126+00:00 app[web.1]: java.lang.NullPointerException: null
2017-09-18T00:41:51.488128+00:00 app[web.1]: 	at com.newrelic.agent.Transaction.checkExpireTracedActivities(Transaction.java:2262) ~[newrelic.jar:3.42.0]
2017-09-18T00:41:51.488129+00:00 app[web.1]: 	at com.newrelic.agent.Transaction.cleanUp(Transaction.java:1096) ~[newrelic.jar:3.42.0]
2017-09-18T00:41:51.488129+00:00 app[web.1]: 	at com.newrelic.agent.TransactionService.processQueue(TransactionService.java:81) ~[newrelic.jar:3.42.0]
2017-09-18T00:41:51.488130+00:00 app[web.1]: 	at com.newrelic.agent.TransactionService$1.run(TransactionService.java:69) [newrelic.jar:3.42.0]
2017-09-18T00:41:51.488131+00:00 app[web.1]: 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144-cedar14]
2017-09-18T00:41:51.488131+00:00 app[web.1]: 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_144-cedar14]
2017-09-18T00:41:51.488132+00:00 app[web.1]: 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_144-cedar14]
2017-09-18T00:41:51.488133+00:00 app[web.1]: 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_144-cedar14]
2017-09-18T00:41:51.488134+00:00 app[web.1]: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144-cedar14]
2017-09-18T00:41:51.488135+00:00 app[web.1]: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144-cedar14]
2017-09-18T00:41:51.488136+00:00 app[web.1]: 	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144-cedar14]
2017-09-18T00:42:21.488881+00:00 app[web.1]: Sep 18, 2017 00:42:21 +0000 [43 24] com.newrelic.agent.TransactionService WARN: Exception processing async update queue.
2017-09-18T00:42:21.488896+00:00 app[web.1]: java.lang.NullPointerException: null
2017-09-18T00:42:21.488897+00:00 app[web.1]: 	at com.newrelic.agent.Transaction.checkExpireTracedActivities(Transaction.java:2262) ~[newrelic.jar:3.42.0]
2017-09-18T00:42:21.488899+00:00 app[web.1]: 	at com.newrelic.agent.Transaction.cleanUp(Transaction.java:1096) ~[newrelic.jar:3.42.0]
2017-09-18T00:42:21.488899+00:00 app[web.1]: 	at com.newrelic.agent.TransactionService.processQueue(TransactionService.java:81) ~[newrelic.jar:3.42.0]
2017-09-18T00:42:21.488900+00:00 app[web.1]: 	at com.newrelic.agent.TransactionService$1.run(TransactionService.java:69) [newrelic.jar:3.42.0]
2017-09-18T00:42:21.488900+00:00 app[web.1]: 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144-cedar14]
2017-09-18T00:42:21.488901+00:00 app[web.1]: 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_144-cedar14]
2017-09-18T00:42:21.488902+00:00 app[web.1]: 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_144-cedar14]
2017-09-18T00:42:21.488903+00:00 app[web.1]: 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_144-cedar14]
2017-09-18T00:42:21.488904+00:00 app[web.1]: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144-cedar14]
2017-09-18T00:42:21.488904+00:00 app[web.1]: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144-cedar14]

Start up log

2017-09-18T00:29:04.568134+00:00 app[web.1]: Sep 18, 2017 00:29:04 +0000 [43 5] com.newrelic.agent.Agent INFO: JVM is shutting down
2017-09-18T00:29:04.581202+00:00 app[web.1]: ~        _            _ 
2017-09-18T00:29:04.581204+00:00 app[web.1]: ~  _ __ | | __ _ _  _| |
2017-09-18T00:29:04.581205+00:00 app[web.1]: ~ | '_ \| |/ _' | || |_|
2017-09-18T00:29:04.581206+00:00 app[web.1]: ~ |  __/|_|\____|\__ (_)
2017-09-18T00:29:04.581206+00:00 app[web.1]: ~ |_|            |__/   
2017-09-18T00:29:04.581207+00:00 app[web.1]: ~
2017-09-18T00:29:04.581208+00:00 app[web.1]: ~ play! 1.5.x-1.4.4-86-gb720d997, https://www.playframework.com
2017-09-18T00:29:04.581209+00:00 app[web.1]: ~ framework ID is prod
2017-09-18T00:29:04.581209+00:00 app[web.1]: ~
2017-09-18T00:29:04.581210+00:00 app[web.1]: ~ Ctrl+C to stop
2017-09-18T00:29:04.581211+00:00 app[web.1]: ~ 
2017-09-18T00:29:04.581212+00:00 app[web.1]: ~ using java version "1.8.0_144-cedar14"
2017-09-18T00:29:08.830701+00:00 heroku[web.1]: Starting process with command `play run --http.port=19653 --%prod -Dprecompiled=true`
2017-09-18T00:29:10.360087+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true
2017-09-18T00:29:10.949880+00:00 app[web.1]: Sep 18, 2017 00:29:10 +0000 [43 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/app/newrelic/./newrelic.yml"
2017-09-18T00:29:11.147062+00:00 app[web.1]: Sep 18, 2017 00:29:11 +0000 [43 1] com.newrelic INFO: Agent Host: 25157eb1-716a-4bbf-87f9-64794bbaca82.prvt.dyno.rt.heroku.com IP: 172.16.68.194
2017-09-18T00:29:11.147298+00:00 app[web.1]: Sep 18, 2017 00:29:11 +0000 [43 1] com.newrelic INFO: New Relic Agent v3.42.0 is initializing...
2017-09-18T00:29:11.986406+00:00 app[web.1]: Sep 18, 2017 00:29:11 +0000 [43 1] com.newrelic.agent.deps.org.reflections.Reflections WARN: given scan urls are empty. set urls in the configuration
2017-09-18T00:29:12.105621+00:00 app[web.1]: Sep 18, 2017 00:29:12 +0000 [43 1] com.newrelic.agent.deps.org.reflections.Reflections INFO: Reflections collected metadata from input stream using serializer com.newrelic.agent.util.EncodingAwareJsonSerializer
...
2017-09-18T00:29:16.481205+00:00 app[web.1]: Sep 18, 2017 00:29:16 +0000 [43 1] com.newrelic.agent.RPMServiceManagerImpl INFO: Configured to connect to New Relic at collector.newrelic.com:443
2017-09-18T00:29:16.520785+00:00 app[web.1]: Sep 18, 2017 00:29:16 +0000 [43 1] com.newrelic INFO: Setting audit_mode to false
2017-09-18T00:29:16.521015+00:00 app[web.1]: Sep 18, 2017 00:29:16 +0000 [43 1] com.newrelic INFO: Setting protocol to "https"
2017-09-18T00:29:18.298990+00:00 app[web.1]: Sep 18, 2017 00:29:18 +0000 [43 1] com.newrelic.agent.config.ConfigServiceImpl INFO: Configuration file is /app/newrelic/./newrelic.yml
2017-09-18T00:29:18.316894+00:00 app[web.1]: Sep 18, 2017 00:29:18 +0000 [43 1] com.newrelic INFO: New Relic Agent v3.42.0 has started
2017-09-18T00:29:18.317034+00:00 app[web.1]: Sep 18, 2017 00:29:18 +0000 [43 1] com.newrelic INFO: Agent class loader: com.newrelic.bootstrap.BootstrapAgent$JVMAgentClassLoader@5680a178
2017-09-18T00:29:18.317408+00:00 app[web.1]: Sep 18, 2017 00:29:18 +0000 [43 1] com.newrelic INFO: Premain startup complete in 7,812ms
2017-09-18T00:29:18.614696+00:00 app[web.1]: Sep 18, 2017 12:29:18 AM play.Logger warn
2017-09-18T00:29:18.614706+00:00 app[web.1]: WARNING: Cannot replace USE_JAVASCRIPT_SOURCE in configuration (useJavaScriptSource=${USE_JAVASCRIPT_SOURCE})
2017-09-18T00:29:19.647918+00:00 app[web.1]: :: loading settings :: url = jar:file:/app/.play/framework/lib/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
2017-09-18T00:29:21.671444+00:00 app[web.1]: 00:29:21,671 WARN  ~ Cannot replace USE_JAVASCRIPT_SOURCE in configuration (useJavaScriptSource=${USE_JAVASCRIPT_SOURCE})
2017-09-18T00:29:28.877626+00:00 heroku[web.1]: State changed from starting to up
2017-09-18T00:29:33.312946+00:00 app[web.1]: Sep 18, 2017 00:29:33 +0000 [43 27] com.newrelic INFO: Host name is 25157eb1-716a-4bbf-87f9-64794bbaca82.prvt.dyno.rt.heroku.com, display host 25157eb1-716a-4bbf-87f9-64794bbaca82.prvt.dyno.rt.heroku.com for application ptasocial-production
2017-09-18T00:29:33.759263+00:00 app[web.1]: Sep 18, 2017 00:29:33 +0000 [43 27] com.newrelic INFO: Collector redirection to collector-214.newrelic.com:443
2017-09-18T00:29:34.205112+00:00 app[web.1]: Sep 18, 2017 00:29:34 +0000 [43 27] com.newrelic INFO: Agent run id: 350199238637830090
2017-09-18T00:29:34.205569+00:00 app[web.1]: Sep 18, 2017 00:29:34 +0000 [43 27] com.newrelic INFO: Agent 43@25157eb1-716a-4bbf-87f9-64794bbaca82.prvt.dyno.rt.heroku.com/ptasocial-production connected to collector.newrelic.com:443
2017-09-18T00:29:34.205751+00:00 app[web.1]: Sep 18, 2017 00:29:34 +0000 [43 27] com.newrelic INFO: Reporting to: https://rpm.newrelic.com/accounts/1418591/applications/28965977
2017-09-18T00:29:34.217377+00:00 app[web.1]: Sep 18, 2017 00:29:34 +0000 [43 27] com.newrelic INFO: Using RUM version 1044 for application "ptasocial-production"
2017-09-18T00:29:34.217453+00:00 app[web.1]: Sep 18, 2017 00:29:34 +0000 [43 27] com.newrelic INFO: Real user monitoring is enabled with auto instrumentation for application "ptasocial-production"
2017-09-18T00:42:21.488905+00:00 app[web.1]: 	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144-cedar14]

New Relic edit

  • I want this, too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.


#2

Hi, just an update. I tried previous versions until the problem went away and I had to go right back to v3.39.1 before the problem was fixed. So looks like something was introduced in v.3.40 that breaks Play1 (1.5) apps. In addition to the exception I did experience some strange application behaviour but not 100% sure they are related. Hopefully this can be resolved otherwise no further upgrades for me.

thx,
Chris


#3

Hey there @tazmaniax - thanks for doing a little of your own digging there, and I’m sorry that you’re having a tough time making this work. Lookin at our docs, it seems that we don’t explicitly support Play 1.5:

https://docs.newrelic.com/docs/agents/java-agent/getting-started/compatibility-requirements-java-agent

Specifically, we support:

  • Play 1.2.4
  • Play 2.0.3 to 2.5.x

It’s always possible that our agent will work in an unsupported environment, but we can’t guarantee or support it. Let me know if that sounds right or if we should dig a little further.


#4

hi @hross, thanks for getting back. That’s very interesting about official support. I haven’t had any problems with upgrades to the NewRelic Java agent so far so just thought NewRelic was keeping track of successive updates to Play1. What is the likelihood NewRelic would provide official support for the latest Play1 version and if that’s not likely can the community get involved to fill the gap?


#5

Hey @tazmaniax -

First - let me just say that I love your user name.

Second - I can’t speak to when our product team will be able to officially support Play v. 2.6.5 (the latest). We do make updates to the Java agent regularly, so be on the lookout. In the meantime, I can make this post a Feature Request so that you and others can vote for an inclusion of Play 2.6.

How does that sound?


#6

It would be great to newrelic to support Play 2.6.x, we lost most of our apps’ monitoring after update them.


#7

Hi @LeonhardtDavid , supporting Play 2.6.5 is already in our roadmap. But we don’t have an ETA yet. You can keep an eye on our release notes about the latest update in the agent: https://docs.newrelic.com/docs/release-notes/agent-release-notes/java-release-notes


#8

Hey @hross, thx for your earlier comments. My request for support is for Play1 not for Play2 - despite the similarity in name they really are different products. Migrating across is not trivial and I don’t have any plans of moving my application off of Play1 any time soon. I would definitely encourage you to support Play1 (latest version is v1.5), which is still being used by lots of organisations. I would be happy to assist. Otherwise I guess I better start looking for another monitoring tool. Thanks for your help.


#9

Thanks @tazmaniax - Sorry I got that version wrong in my last reply. I hear that you are looking for 1.5 support. I’m adding this thread to our Feature Ideas category and adding a poll to it. You can encourage others to come vote and share their use cases with our product team.


#10

@hross I know we’re a year later and I haven’t done very well with getting more votes but still interested to get up to date Play v1.5 support. I’ve updated to v4.8.0 but still getting the following exceptions in the Heroku log…

2018-11-28T06:43:34.313959+00:00 app[web.2]: Nov 28, 2018 06:43:34 +0000 [50 24] com.newrelic.agent.TransactionService WARN: Exception processing async update queue.
2018-11-28T06:43:34.313975+00:00 app[web.2]: java.lang.NullPointerException: null
2018-11-28T06:43:34.313978+00:00 app[web.2]: at com.newrelic.agent.Transaction.checkExpireTracedActivities(Transaction.java:2398)
2018-11-28T06:43:34.313980+00:00 app[web.2]: at com.newrelic.agent.Transaction.cleanUp(Transaction.java:1234)
2018-11-28T06:43:34.313981+00:00 app[web.2]: at com.newrelic.agent.TransactionService.processQueue(TransactionService.java:82)
2018-11-28T06:43:34.313983+00:00 app[web.2]: at com.newrelic.agent.TransactionService$1.run(TransactionService.java:70)
2018-11-28T06:43:34.313984+00:00 app[web.2]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2018-11-28T06:43:34.313985+00:00 app[web.2]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
2018-11-28T06:43:34.313987+00:00 app[web.2]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
2018-11-28T06:43:34.313988+00:00 app[web.2]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
2018-11-28T06:43:34.313990+00:00 app[web.2]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2018-11-28T06:43:34.313991+00:00 app[web.2]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2018-11-28T06:43:34.313992+00:00 app[web.2]: at java.lang.Thread.run(Thread.java:748)

#11

Hi @tazmaniax,

Thanks for the update. I’m more than happy to investigate this further.

Looking at the log snippets below, I wonder if this has something to do with some deadlock occurrence, where the agent locks the current transaction on some threads, while linking both threads due to a mismatch:

2018-11-28T06:43:34.313984+00:00 app[web.2]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2018-11-28T06:43:34.313985+00:00 app[web.2]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
2018-11-28T06:43:34.313987+00:00 app[web.2]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
2018-11-28T06:43:34.313988+00:00 app[web.2]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
2018-11-28T06:43:34.313990+00:00 app[web.2]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2018-11-28T06:43:34.313991+00:00 app[web.2]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

I can’t say for sure where this would lead us to, but I’m happy to investigate further :smile:. As next steps, kindly collect some thread dumps(using Jstack) from the JVM while the agent is running, and check if there are any deadlocks. You can search for either of these line statements :

waiting to lock monitor .....,

or

java.lang.Thread.State: BLOCKED (on object monitor) 
at com.newrelic.agent.Transaction.getCrossProcessTransactionState.....

If you do come across this piece of information. Kindly share a short snippet of the thread information on the community post for further review.

Thanks.