New Relic One, Container Data, Logs Not Appearing in the Explorer -> Containers

Team, while testing new ways of monitoring docker containers, I started to use New Relic Infrastructure Agent to monitor not only the underlying host but also added the Docker Integration and tested it for both Linux and Windows machines.
I have successfully forwarded the logs and metrics from these servers and containers, however we failed to understand why the logs are not appearing or being associated with the specific container.

To answer a few questions ahead:

  • Yes, the new relic infrastructure agent is currently running with root permissions
  • Yes, we have installed the log forwarding solution, FluentBit to allow logs to be streamed to new relic one. Although NR Infra agent is handling it itself without the necessity of having FluentBit service running. So, for instance the log entry below was forwarded from NRI and our understanding is the it should also appear in the Container interface, but we are not able to perform the correct association with the container because it does not contain the container name and other relevant docker information.

Below you can find the logging.yml used in this environment deployed at /etc/newrelic-infra/logging.d path. As you can see we are basically tailing the /var/lib/docker/containers//.log to fetch all logs from all running containers at once. And we were successful at it, however we want to understand more on how to associate the logs to the correct container as it will be an useful feature in case we move forward with New Relic solution.

logs:
  - name: alternatives.log
    file: /var/log/alternatives.log
    attributes:
      logtype: linux_alternatives
  - name: cloud-init.log
    file: /var/log/cloud-init.log
    attributes:
      logtype: linux_cloud-init
  - name: auth.log
    file: /var/log/auth.log
    attributes:
      logtype: linux_auth
  - name: dpkg.log
    file: /var/log/dpkg.log
    attributes:
      logtype: linux_dpkg
      #- name: syslog
      #file: /var/log/syslog
      #attributes:
      #logtype: linux_syslog
  - name: containers
    file: /var/lib/docker/containers/*/*.log
    attributes:
      application: dockercontainers
      maintainer: 
      image: docker.imageName

  - name: newrelic-cli.log
    file: /root/.newrelic/newrelic-cli.log
    attributes:
      newrelic-cli: true
      logtype: newrelic-cli

Following is our docker-config.yml from our /etc/newrelic-infra/integrations.d path.

integrations:
  - name: nri-docker
    when:
      feature: docker_enabled
      file_exists: /var/run/docker.sock
    interval: 15s
  - name: nri-docker
    when:
      feature: docker_enabled
      env_exists:
        FARGATE: "true"
    interval: 15s

Thanks for the support.

Regards

Hello @user2505,

Welcome to the community!

While your question is a bit out of my scope with logs, I am going to loop in one of our engineers to look over this and provide better insight. We appreciate the incredibly detailed post as all of this information will help the team find a solution for you. Please let us know if there is anything else we can help with as well.

I hope you have a great day!

Hi user2505, thanks for writing in to our community.

AFAIK in order to tag or identify the logs so they can be associated to the source container, one would need to customize the log driver output, per this Docker document.

Using the log tag option, append --log-opt tag="{{.Name}}" to the run command so that the log driver will decorate the logs with container name.

2 Likes

Hi there,

I was in a similar situation to the above but was able to get the logs associated with the containers. However, I’ve noticed an issue that if a container shares the same name (which in our case will happen across hosts), the logs will appear in all of the container logs as opposed to the one with the correct container id. What’s odd is if I filter based on the host hostname instance I’ll only get container logs related to that host, although they’ll be mushed together (unless filtered by an attribute).
The fluentd filter being used is -

<filter docker.*>
  @type record_transformer
  enable_ruby
  <record>
      hostname "#{Socket.gethostname}"
      service_name ${tag}
      containerName ${tag[7..-1]}
  </record>
</filter>

It would be ideal if we could have the logs matched directly to the unique container - is there a tag I can attach to help facilitate this? Attributes such as the container id are already appearing in New Relic so I’m a bit surprised it’s not matching that way.

Hello @ada1 ,

My colleague Tony gave a great response. While containers can share a name they will always have a unique container id. The container id to the specific container is nestled in the file path of the log file.

The docker documentation gives a clear overview of logging from containers. The path will contain the container id. For example:
/var/lib/docker/containers/{ContainerId}/{ContainerId}-json.log.

You can leverage the filePath metadata in your log message to accomplish your goal. We capture the full path of the log as filePath. It will have the unique id for that particular container nested within.

If you want to add any further tags, you can reference the Customize log driver output

Hope this helps.