Parsing attributes from log line

I’m using java agent to forward logs to NewRelic and ingestion works fine but I’d like to parse custom property which is defined before log level but in new relic I can’t see this at all in message attribute so I’d like to know how can I parse this or make this visible in NewRelic since my understanding is that parsing rules can be applied only to the message attribute and my guess is that during ingestion that part of the log is lost somewhere.

Pattern: %d{HH:mm:ss.SSS} [%thread-id] [component] [id1] [id2] %-5level %logger{35} - %msg %n

Log line example: 09:45:55.130 [pool-38-thread-1] [Management] [81e5c1cb-c9ec-4457-b657-2b250b7d2e55] [f01c7027-568f-4380-acd4-56e5c1315114] DEBUG com.zaxxer.hikari.pool.HikariPool - keepalive: connection org.postgresql.jdbc.PgConnection@7333f8e is alive

Parsed log details in new relic:

{
  "entity.guid": "MzY2MDk0NnxBUE18QVBQTElDQVRJT058MTA1NzI0MDQxMQ",
  "entity.guids": "MzY2MDk0NnxBUE18QVBQTElDQVRJT058MTA1NzI0MDQxMQ",
  "hostname": "aa2aab97c275",
  "level": "DEBUG",
  "logger.fqcn": "ch.qos.logback.classic.Logger",
  "logger.name": "com.zaxxer.hikari.pool.HikariPool",
  "message": "keepalive: connection org.postgresql.jdbc.PgConnection@7333f8e is alive",
  "newrelic.logPattern": " keepalive: connection <*> is alive",
  "newrelic.source": "logs.APM",
  "thread.id": 61,
  "thread.name": "pool-38-thread-1",
  "timestamp": 1668678833785
}

The outcome I’d like to have:

{
  "entity.guid": "MzY2MDk0NnxBUE18QVBQTElDQVRJT058MTA1NzI0MDQxMQ",
  "entity.guids": "MzY2MDk0NnxBUE18QVBQTElDQVRJT058MTA1NzI0MDQxMQ",
  "hostname": "aa2aab97c275",
  "level": "DEBUG",
  "logger.fqcn": "ch.qos.logback.classic.Logger",
  "logger.name": "com.zaxxer.hikari.pool.HikariPool",
  "message": "keepalive: connection org.postgresql.jdbc.PgConnection@7333f8e is alive",
  "newrelic.logPattern": " keepalive: connection <*> is alive",
  "newrelic.source": "logs.APM",
  "thread.id": 61,
  "thread.name": "pool-38-thread-1",
  "timestamp": 1668678833785,
  "component": "Management",
  "id1":"81e5c1cb-c9ec-4457-b657-2b250b7d2e55",
  "id2": "f01c7027-568f-4380-acd4-56e5c1315114"
}

What would be the best way to proceed with in order to parse and store component, id1 and id2 as attributes of that log line.

Thanks in advance!

P.S. NewRelic is pretty new to me so excuse me if I missed some obvious answer for this.

Hi @lovro

Thanks for reaching out, I hope you are well.

Congrats on your first post in the community, Whoop!

It sounds like you are having some trouble with parsing, please see the New Relics docs on Parsing log data as they should help you here.

I hope this was helpful, should you have any questions or updates please do reach out.

Also if this helped resolve the issue please consider selecting the Solution ( How to Use the Official Solution Checkbox )option below this post.

Hi @dcody

As you can see from the given example message is already parsed either by java agent or by NewRelic platform and therefore I can’t use log parsing functionality if API is not newrelic.source. When sending logs using API I get the full log message in message field and I’m able to parse it from there.

Hey @lovro,

Thanks for your patience here.

Our Java APM agent forwards and formats the logs using whichever logging framework is being used in your application and uses a pretty rigid log format. This means that custom attributes won’t be possible using this method.

There are a few workarounds:

  1. One option is to disable log forwarding from the agent and use FluentBit or another log forwarder instead. You can still use the APM agent to decorate the logs, but then when you are forwarding the logs using FluentBit, you can add attributes there.
    https://docs.newrelic.com/docs/logs/logs-context/java-configure-logs-context-all#2-decorate

  2. Another option would be to add those values to your message itself by adding them as tags or appending them to the end of the message. From there, you would be able to create a custom parsing rule to parse them out.

Thanks!

1 Like