Is it possible to fetch spark cluster/driver logs (stdout, stderr, log4j) into New Relic using Infrastructure agent

I have a spark streaming application running on databricks, Is it possible to fetch the spark cluster/driver logs into new relic using infrastructure agent and is it possible to parse these log files in new relic. Please let me know ASAP

Hi @swetha.kadiyala

Thanks for reaching out, I hope you are well.

Im not sure what “spark” is however below I discuss how to forward logs and parse them.

In short you can use the infra agent to forward the logs in question. We even have these great docs which discuss how to Forward your logs using the infrastructure agent.

For parsing logs, the best steps I personally found best are those using Grok. I did a little digging located some New Relic docs that guide user through parsing logs. Please see the doc, for guidance.

I hope this was helpful, should you have any questions or updates please do reach out.

Also if this helped resolve the issue please consider selecting the Solution ( How to Use the Official Solution Checkbox )option below this post.