I’m trying to create a NRQL query that returns the latest node status in a k8s cluster based on the NodeReady and NodeNotReady events.
The closest I got to that is:
SELECT latest(event.reason) from InfrastructureEvent where event.reason in (‘NodeNotReady’,‘NodeReady’) facet event.involvedObject.name since 356 days ago
But sometimes it returns “NodeNotReady” for nodes that are actually ready and if I query the node specific data I can see the latest event.reason is indeed “NodeReady” when sorted by event.lastTimestamp.
I think “latest()” is using the reporting timestamp and not the actual event timestamp.
I’m attaching a few screenshots filtered for a specific node that was showing this behavior a few minutes ago. I’ve highlighted some areas that I think will help clarify the issue.
Query showing the “NodeNotReady” result for a node that is Ready:
Another query showing the NodeReady and NodeNotReady events for that same node with default sorting. Note that the two most recent events have the same timestamp but their event.lastTimetamp differs:
And the same query sorted by event.lastTimestamp desc:
New Relic Edit
- I want this too
- I have more info to share (reply below)
- I have a solution for this
We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.