Wrong timestamp and incorrect log order

Hi,
We stream the application log from ECS to New Relic with the flow as below :
AWS ECS > AWS FireLens with Fluent Bit as a sidecar container> AWS Kinesis Data FireHose > New Relic HTTP endpoint

We can see the logs coming up to New Relic. But the problem is:

  1. The timestamp column in New Relic Logs is different from the message log’s timestamp. The value in the timestamp column that we need is supposed to be timestamp in the message log.

  2. The logs are in incorrect order. If you see the picture below, the millisecond (4th column) shows it all (jumbled order)

How to solve this issue?

Hi @ghoran @nmcnamara
Please have a look at this issue. Thankyou

Hi, as an update, we stream our logs with additional attribute called “time”. So this is the actual timestamp that we need.

My questions are :

  1. If we choose time frame, it will reflect to new relic timestamp. Can we make it reflect to “time” attribute?

  2. How can we order the logs based on “time” order?

  3. If none of my question are doable, how can we use the actual timestamp instead of new relic timestamp?

Please take a look on our ticket. We are facing this issue for so long and didn’t get any response from this community. This is a huge problem that we faced and made us difficult to debug the logs.

@ghoran @nmcnamara

Hi,
We stream the application log from ECS to New Relic with the flow as below :
AWS ECS > AWS FireLens with Fluent Bit as a sidecar container> AWS Kinesis Data FireHose > New Relic HTTP endpoint

We can see the logs coming up to New Relic. But the problem is:

The timestamp column in New Relic Logs is different from the message log’s timestamp. The value in the timestamp column that we need is supposed to be timestamp in the message log (actual timestamp) instead of ingestion time

What we did :
After finding the above issue, we stream our logs with additional attribute called “time”. So this is the actual timestamp that we need.

My questions are :

  1. If we choose time frame, it will reflect to new relic timestamp. Can we make it reflect to “time” attribute?

  2. How can we order the logs based on “time” order?

  3. If none of my question are doable, how can we use the actual timestamp instead of new relic timestamp?

Please take a look on our ticket. We are facing this issue for so long and didn’t get any response from this community, from previous ticket (Wrong timestamp and incorrect log order). This is a huge problem that we faced and made us difficult to debug the logs.

Hi @jaswenny1 :wave: ! A ticket has been created on your behalf. Please keep an eye out for an email from one of our support agents, they will assist you with your request - Thank you!

Hi, Thank you!
But, it’s been 2 days, and none of the support agents emailed me.
Please help since this is a big bottleneck for us.

Hi @jaswenny1! Our team should reach out shortly to assist!

It seems that it is not only the ECS logs to Firehose which have problems of ingestion of the timestamp … I have the same problem with the WAF logs ==> Timestamp of logs are wrong when logs are send from AWS Kinesis Data Firehose

For my issue, New Relic team said that there is a bug in Firehose-NewRelic integration. That’s why we keep getting wrong timestamp.

2 Likes

oh! Thank @jaswenny1

A bug on the NewRelic integration side and not AWS FH, is that right? Did they give you a possible resolution date?