Entity.Guid not searchable with Serilog, .NET Core in Docker and Infrastructure agent

Support: Telemetry Data Platform (TDP) Log Management
,
#1

I am running a .NET Core 3.1 application in a docker container. The application uses Serilog and I set up the logging like this:

var loggerConfig = new LoggerConfiguration()
            .MinimumLevel.Information()
            .MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
            .MinimumLevel.Override("System", LogEventLevel.Error)
            .Enrich.WithThreadId()
            .Enrich.WithThreadName()
            .Enrich.WithNewRelicLogsInContext()
            .WriteTo.Console(new NewRelicFormatter());

        Log.Logger = loggerConfig.CreateLogger();

The logs show up in the container logs like this:

{"log":"{\"timestamp\":1588153750962,\"log.level\":\"Information\",\"message.template\":\"Import finished; took 188 ms.\",\"message\":\"Import finished; took 188 ms.\",\"Message.Properties.ThreadId\":37,\"entity.name\":\"DataImporter\",\"entity.type\":\"SERVICE\",\"entity.guid\":\"MjQ5OTM5N3xBUE18QVBQTElDQVRJT058MzU3OTIwNDg\",\"hostname\":\"658d5e66a89d....\"}\n","stream":"stdout","time":"2020-04-29T09:49:10.96280617Z"}
{"log":"{\"timestamp\":1588153760772,\"log.level\":\"Information\",\"message.template\":\"Import started...\",\"message\":\"Import started...\",\"Message.Properties.ThreadId\":63,\"entity.name\":\"DataImporter\",\"entity.type\":\"SERVICE\",\"entity.guid\":\"MjQ5OTM5N3xBUE18QVBQTElDQVRJT058MzU3OTIwNDg\",\"hostname\":\"658d5e66a89d....\"}\n","stream":"stdout","time":"2020-04-29T09:49:20.773183208Z"}
{"log":"{\"timestamp\":1588153760947,\"log.level\":\"Information\",\"message.template\":\"Import finished; took 174 ms.\",\"message\":\"Import finished; took 174 ms.\",\"Message.Properties.ThreadId\":63,\"entity.name\":\"DataImporter\",\"entity.type\":\"SERVICE\",\"entity.guid\":\"MjQ5OTM5N3xBUE18QVBQTElDQVRJT058MzU3OTIwNDg\",\"hostname\":\"658d5e66a89d....\"}\n","stream":"stdout","time":"2020-04-29T09:49:20.948214369Z"}

And they are shipped through New Relic Infrastructure agent to NewRelic. I do see them like this:

grafik
grafik1793Ă—323 36.3 KB

But I cannot query them with

entity.guid:MjQ5OTM5N3xBUE18QVBQTElDQVRJT058MzU3OTIwNDg

only with “fulltext” search like

MjQ5OTM5N3xBUE18QVBQTElDQVRJT058MzU3OTIwNDg
#2

Hi @maximilian.beller ! This is an interesting situation. The main issue is that the original JSON log is getting wrapped inside of that {"log": key, which prevents our ingest from parsing those values correctly. We expect the data to just look like:

{\"timestamp\":1588153750962,\"log.level\":\"Information\",\"message.template\":\"Import finished; took 188 ms.\",\"message\":\"Import finished; took 188 ms.\",\"Message.Properties.ThreadId\":37,\"entity.name\":\"DataImporter\",\"entity.type\":\"SERVICE\",\"entity.guid\":\"MjQ5OTM5N3xBUE18QVBQTElDQVRJT058MzU3OTIwNDg\",\"hostname\":\"658d5e66a89d....\"}

However, I’m not sure what is wrapping that log line with additional JSON. Does anything stand out to you based on your environment?

As a test, would it be possible to configure Serilog to write directly to a file (instead of console) then exec into that container and grab a couple of the resulting log lines? This just narrows down the list of possible causes.

Let me know what you think! Cheers.

#3

Hi,

the log files look exactly as you said it is expected when Serilog writes to a file in the container. Also, it looked exactly like that when I docker logs. The wrapping into “log” only happens when docker itself persists the data via the JSON File logging driver to disk.

Is there someway that NewRelic can unwrap the data? I think that many users using docker will have this requirement, no?

#4

Hi @maximilian.beller - Thanks for that information, you’re right that we should be unwrapping these lines. I’m going to open up a ticket with you for this, so that we can dig into more specific information on your account. You should get an email about that shortly!

1 Like
#5

Can you please help me with the setup of serilog with newrelic. Can this be only configured in web.config? Or do I need to follow all the steps mentioned in: https://docs.newrelic.com/docs/logs/enable-log-management-new-relic/logs-context-net/net-configure-serilog

#6

Hi skutti!

You will want to follow the steps in the documentation. You can configure Serilog using the Serilog extension or with the file configuration options. You will want to make sure you are also using one of our compatible log forwarders.

#7

Unable to initialize the New Relic .NET Agent log file. Please make sure the logs directory is writeable.

I am getting above error and logs are not forward to Newrelic. Can you share other configurations

#8

Hi @ashok.nakulan

Thanks for reaching out!

Can you provide a permalink to where this occurring if possible. Please note that only New Relics will be able access this!

This will allow us to gain a full understanding of the ins & outs of the issue!