Logs JSON parsing

Hi folks!

Quick question about an odd behavior we’re noticing when using New Relic Logs—while we’re sending properly-formatted JSON structured logs, those somehow aren’t automatically parsed by New Relic, despite what the official documentation states. Our setup is the following: we have a Rails 6 app hosted on Heroku. The logs are forwarded to New Relic via a Syslog drain, and they’re formatted by our app using both the Lograge gem and the New Relic Agent via the dedicated formatter.

Everything seems to work fine up until the logs actually reach New Relic—here’s how they’re received:


As you can see, the JSON payload is treated as a string and not parsed, even though it’s perfectly valid and within the character limit. Can you advise? I’d rather avoid setting up complex Grok rules to parse it myself if I can avoid it.

2 Likes

Hi @tech-owner,
There’s a similar thread on this subject here: Parsing rules seems not applied to logs

That thread specifically mentions a known bug where custom parsing isn’t working for Heroku syslog drains. I’m checking on my end to see if this bug applies to your situation and extends to our default JSON parsing. From what you’re seeing it looks like the answer is ‘yes’, but I want to verify before declaring it so. I’ll post an update when I find out more.

1 Like

Thank you @lchapman — let me know!

Hi @tech-owner,
I verified on my end with the product team and unfortunately our default JSON parsing is also not working when logs are sent to us from Heroku. That JSON parsing feature is part of the bug mentioned in the post I mentioned earlier and will be fixed with the rest of the parsing issues from Heroku. The rough timeframe of next month for a fix still stands.

You could try creating a custom parsing rule in our UI…I know the issue above mentioned it didn’t work but it wouldn’t hurt to try a simple grok pattern, ie %{GREEDYDATA:TestParse}. That pattern would copy all of the heroku data from the message attribute into the TestParse attribute. I did it for a regular syslog data feed (not from Heroku) and was able to parse the syslog data further. Doc link to our parsing UI here: Parsing log data | New Relic Documentation

Hope that helps.

Understood—thank you @lchapman !