Forwarding systemd logs with NRIA - message field not recognized

Support: Telemetry Data Platform (TDP) Log Management
#1

I’ve installed the NRIA and set up log forwarding from running Docker containers using systemd. The Docker log driver is journald. The log forwarding works, but my log events are missing the message field.

I am outputting JSON logs, and when I look at the entire log event, I can see the JSON fields being parsed properly, e.g. MESSAGE.message, MESSAGE.timestamp, etc.

But it looks like NR is looking for the message field, not MESSAGE, so the message column is empty for all of my logs. What am I doing wrong here?

#2

Hey there @glib,

I hope you are well! I see that you are missing some information in your logs in New Relic. I think this may help with that and I would recommend possibly running a query to see if there are any errors related to logging: No log data appears in the UI | New Relic Documentation.

I do apologize that I do not have a direct answer as your question is a bit out of my scope but I am looping in an expert from our logging team to provide more insight towards a resolution for you. I appreciate your patience while we work on supporting this.

Please let us know if there is anything else we can help with, we are more than happy to assist!

#3

Hello @glib , I set up a quick reproduction with the infra agent, and Docker using the journald logging driver on Linux. I also used the systemd.yml.example as a basis and ended up with this for logging.d/systemd.yml:

logs:
  - name: systemd-docker
    systemd: docker

This results in incoming logs from my Docker containers, and the message is being parsed normally - shows up as message in the Log events and the Logging UI.

Is your application specifically outputting MESSAGE as a top level attribute? Our automatic JSON parsing is designed to work with a message field which contains a valid JSON object. If the logs from the container are in the form MESSAGE.attribute1, MESSAGE.attribute2 etc. things might not work as expected.

1 Like
#4

Yes, that’s how I have the agent set up as well. The logs do show up.

Is your application specifically outputting MESSAGE as a top level attribute?

No, it’s the Infra agent that’s putting the JSON that my app is outputting into a MESSAGE field.

I’m using Python with NewRelicConextFormatter for my log messages, so it’s New Relic that’s handling the log format.

#5

@glib Thanks for clarifying that this is the Python agent generating the logs in question.

Can you post the relevant section of the application code where the logger is being configured, i.e. the equivalent of the example from the Python Logs in Context docs?

Please also post the configuration from the infra agent’s log forwarding component (logging.d/logging.yml) so we can see how it’s being parsed.

Normally, we’d be receiving a JSON formatted message field and parsing it into top level attributes, so I’m not sure where the MESSAGE nesting is happening.