Parsing Message field of syslog-rfc5424

Hi

I am sending logs in syslog-rfc5424 format from an on-premise rsyslog server. The logs are showing up in New Relic.

I have setup a parsing profile, which when testing the GROK pattern, successfully identifies the data I want to parse from the syslog message field. After applying the parsing profile and waiting for a while for new logs to be ingested, I was expecting to see additional attributes with the data parsed from the message field but I do not see any additional attributes.

What is the point of setting up a parsing profile for the syslog message field if I cannot do anything with the data that has been parsed i.e. create a visualisation for dashboards or use it for filtering data.

Am I doing something wrong, configured something incorrectly or is what I am experiencing ‘expected behaviour’?

Many thanks
Darren

Hi @darren.bird

You shouldn’t necessarily need to manually create parsing rules for rfc5424 logs, as we have a built in ruleset for this: Built-in log parsing rulesets | New Relic Documentation - you should just need to specify: logtype = 'syslog-rfc5424' such that, upon ingest, we know what format to parse the data from.

Regardless of if you are manually parsing those logs out, or using the built in rules, I expect that the parsing rules will surface attributes for you to visualise and query.

Can you share a link to the parsing rule you created?

Hi

Parsing of rfc5424 is happening and I can see attributes for app.name, hostname etc that is parsed from the syslog message.

The parsing profile i have setup is to specifically parse just the message field of the syslog message into additional attributes. I want to do this as the message field contains the source and destination IP address as well as port number information.

I cannot see how to share a link to the parsing rule. Instead I have included a screenshot of the parsing rule (including the GROK statement):

Also is a screenshot showing the results of a test parse against my logs:

When clicking on one of my logs in the parsing wizard, the following shows:
image

I was expecting to see the new attributes show in the screenshot above to be available with New Relic when searching/visualising logs.

Look forward to your response and hopeful for a resolution soon.

Thanks
Darren

Hey @darren.bird

I think I see the problem - My understanding is that you have the built in parsing rules working in place, but you’re trying to get additional attributes out of that via a custom parsing rule.

This will not work, as only one parsing rule can be applied to a log line that is ingested. The first that succeeds is the one that is applied.

This is in our documentation here: Parsing Log Data: How it works

Parsing will only be applied once to each log message. If multiple parsing rules match the log, only the first that succeeds will be applied.

You may want to disable the built in rules, and just have one custom parsing rule to pick out everything from the data.

Built in rules are applied based on the logtype attribute, so you should be able to simply take that attribute out of the source data to stop that built in rule applying.

1 Like