In a dashboard, can we filter both logs and metrics at once?

Hi,

I have a dashboard that displays a mix of logs and metrics (CPU, memory, etc).

When looking for ways to create one generic dashboard for our applications, I successfully used this tip to filter the metrics based on the attribute ‘k8s.containerName’. That way, the dashboard displays the CPU for all of our applications in a given environment, but we can filter down to one specific app.

The widget I use to filter has this NRQL:

SELECT average(k8s.container.cpuCoresUtilization) FROM Metric FACET k8s.containerName SINCE 1 day ago WHERE k8s.namespaceName = 'our-dev-environment'

However, when doing so I noticed that the logs disappeared. When looking into it, I noticed that our logs are tagged with the ‘container_name’ attribute, instead of ‘k8s.containerName’! Is that normal?

As a test, I was able to filter our logs using this widget and NRQL:

SELECT average(k8s.container.cpuCoresUtilization) FROM Metric FACET k8s.containerName AS container_name SINCE 1 day ago WHERE k8s.namespaceName = 'our-dev-environment'

The problem is, I would like to have one widget filter both logs and metrics. Is that possible? I’m OK with adding some sort of parsing rule or something similar to add the tag on either side (eg: add ‘container_name’ on metrics).

We’re using NewRelic ‘NRI Bundle’ chart 3.2.11 on Kubernetes 1.22.4.

Thanks

PS: for what it’s worth, I tried using a “double” AS clause in the filter widget, but it doesn’t work because the filter button add an AND clause between the filters, so both the metrics and logs are missing

SELECT average(k8s.container.cpuCoresUtilization) FROM Metric FACET k8s.containerName, k8s.containerName AS 'k8s.containerName', container_name SINCE 1 day ago WHERE k8s.namespaceName = 'dealer-dev'

For what it’s worth, I was able to do it by modifying the logging part to have the same key as the metrics. I did it by adding a custom FluentBit filter to create the needed tag.

The default NewRelic configuration takes the keys under kubernetes and raise them a level (so kubernetes.container_name becomes container_name. Before it does that though, you can use a modify filter to Copy (or Rename) the key kubernetes.container_name to k8s.containerName.

Here’s a partial sample config

newrelic-logging:
  fluentBit:
    config:
      lowDataModeFilters: |
        # Add the default filters here but add the filter below before
        # the default nest & lift filter, or adjust the copy from key

        [FILTER]
            Name modify
            Match kube.*
            Copy kubernetes.container_name k8s.containerName

You can obviously adjust more than just container_name. It would be inconsistent to have tags like k8s.containerName and container_image

2 Likes

Hello @aleveille -

Just wanted to pop in and say thank you so much for sharing the details of what you learned with the rest of the community!