Incorrect log order

Hi team,

I’m using FluentD in K8s to fire logs from our pods to NewRelic. However, log events in NewRelic console seem like not right ordering.
Screenshot 2022-11-18 at 14.50.08
Is this Newrelic bug?

Please help me to resolve this.
Thanks.

Hi @newrelic709

Thanks for reaching out, I hope you are well.

Can you confirm if you are still noting this issue ?

Hi @dcody ,

The issue is existing. Can you give me your recommendation?
Thanks

Hi @newrelic709

I hope you are having a good day.

This support topic is a little outside of my scope, however we see you, and we are working towards finding a solution. I will need to loop in Logs team, an expert from this team will reply here to support this issue.

Hi @newrelic709 ,

@dcody asked me to take a look at this… We accept logs in the order they are sent to us. Is it possible to share a larger screen shot with the timestamp filed showing. As this will tell us the order it has come in. If this field is correct then you may have an issue with your forwarder or firewall.

G

Hi @GlenOFoghlu ,

Please check it as attached files




Hi team,
Any update for this?

Hi @newrelic709

Thanks for reaching out, apologies for the delay here. I do note that @GlenOFoghlu is working on this and aims to reply with his findings tomorrow.

1 Like

The issue here, that I see, is that your logs have a timestamp attached to the Message attribute.

If you can split that up, such that the timestamp is sent as it’s own attribute, and it is in the right format (epoch timestamp, ie: Right now is 1669821959). Your timestamp will overwrite the timestamp you see currently in New Relic.

As it stands, the timestamp attribute that is seemingly incorrect, is the New Relic ingest time, which is set like that because we cannot see the right timestamp to use instead. Splitting that log message up to include the timestamp as it’s own attribute should solve this issue for you.

@newrelic709 Just to add to @RyanVeitch 's answer, here’s our documentation concerning our Log API payload requirements:

https://docs.newrelic.com/docs/logs/log-api/log-event-data/#attributes

Hi @RyanVeitch ,
Why in some cases, it is right, but another is wrong :thinking:

Again this can depend on when the logs get to us. Are you batching your logs, so that you collect 5-10 mins of logs before pushing them to NR? Or similar?

Also, are the logs in your screenshot all from the same service? Or is that a view that may be mismatched by other logs from other sources?

Hi @RyanVeitch
I used default config of fluentbit for logging.
The logs in my screenshot came from one service and the images were sorted is in order as you can saw.