Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Newrelic java agent for Profiling Spark drivers and executors


#1

Hi,

I have been trying to find out if newrelic supports spark job profiling. is there a javaagent that supports profiling spark executors and drivers. if so, is there documentation available for this.

Thanks,
Manaswini


#2

Hi Manaswini,

To be clear, there is only one New Relic Java Agent. While Spark isn’t supported, I’d expect that as it is Java based, you should be able to get some information out of it. I am quite unfamiliar with Spark but I expect that Custom Instrumentation may be required.

Cross Application Tracing also looks like it might be relevant.


#3

Hey @manaswini.sugatoor. I realize this post is a bit dated, however, I recently did some research into instrumenting Apache Spark. Posting some info because I came across this post during my research.

The primary way to get metrics from Spark is through a JMXSink. You could use the Java Agent or Infrastructure Agent to bring in the Spark JMX metrics for the Master, Workers, Drivers, and Executors. For application specific transaction events, you’d need to use Customer Instrumentation. I wrote the following level-up post that goes over the high-level of how to monitor Apache Spark along with a speed through version of implementing JMX extensions with the Java Agent: Monitoring Apache Spark