Feature Idea: New Relic one not receiving logs after Elastic beanstalk integration

Hi there,
I configured “in-context” logging as described in the documentation and have confirmed that everything is installed as described on my instances:

• newrelic-infra is installed
• logging.d folder shows up as expected under /etc/newrelic-infra
• Logging config file is available and configured as specified. Target log files exist and are being written to. Here are the contents of my config file:

logs:
  - name: production.log
    file: /var/app/containerfiles/logs/production.log

  - name: sidekiq.log
    file: /var/app/containerfiles/logs/sidekiq.log

  - name: puma.log
    file: /var/log/puma/puma.log

  - name: access.log
    file: /var/log/nginx/access.log
    attributes:
      logtype: nginx

  - name: error.log
    file: /var/log/nginx/error.log
    attributes:
      logtype: nginx-error

Any ideas as to what else to check?

Running the newrelic-infra service in verbose mode, the following error is shown in the logs:

fluent-bit: error while loading shared libraries: libsystemd.so.0: cannot open shared object file: No such file or directory"

Log snippet below:

time="2020-11-01T06:09:08-06:00" level=debug msg="Creating temp config file for fb sFBLogger." component=integrations.Supervisor content="\n[INPUT]\n    Name tail\n    Path /var/app/containerfiles/logs/production.log\n    Buffer_Max_Size 128k\n    Skip_Long_Lines On\n    Path_Key filePath\n    Tag  rails\n    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db\n\n[INPUT]\n    Name tail\n    Path /var/app/containerfiles/logs/sidekiq.log\n    Buffer_Max_Size 128k\n    Skip_Long_Lines On\n    Path_Key filePath\n    Tag  sidekiq\n    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db\n\n[INPUT]\n    Name tail\n    Path /var/log/puma/puma.log\n    Buffer_Max_Size 128k\n    Skip_Long_Lines On\n    Path_Key filePath\n    Tag  puma\n    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db\n\n[INPUT]\n    Name tail\n    Path /var/log/nginx/access.log\n    Buffer_Max_Size 128k\n    Skip_Long_Lines On\n    Path_Key filePath\n    Tag  nginx-access\n    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db\n\n[INPUT]\n    Name tail\n    Path /var/log/nginx/error.log\n    Buffer_Max_Size 128k\n    Skip_Long_Lines On\n    Path_Key filePath\n    Tag  nginx-errors\n    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db\n\n[INPUT]\n    Name tail\n    Path /var/app/containerfiles/logs/production.log\n    Buffer_Max_Size 128k\n    Skip_Long_Lines On\n    Path_Key filePath\n    Tag  rails\n    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db\n\n[INPUT]\n    Name tail\n    Path /var/app/containerfiles/logs/sidekiq.log\n    Buffer_Max_Size 128k\n    Skip_Long_Lines On\n    Path_Key filePath\n    Tag  sidekiq\n    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db\n\n[INPUT]\n    Name tail\n    Path /var/log/puma/puma.log\n    Buffer_Max_Size 128k\n    Skip_Long_Lines On\n    Path_Key filePath\n    Tag  puma\n    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db\n\n[INPUT]\n    Name tail\n    Path /var/log/nginx/access.log\n    Buffer_Max_Size 128k\n    Skip_Long_Lines On\n    Path_Key filePath\n    Tag  nginx-access\n    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db\n\n[INPUT]\n    Name tail\n    Path /var/log/nginx/error.log\n    Buffer_Max_Size 128k\n    Skip_Long_Lines On\n    Path_Key filePath\n    Tag  nginx-errors\n    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db\n\n[FILTER]\n    Name  record_modifier\n    Match rails\n    Record fb.input tail\n\n[FILTER]\n    Name  record_modifier\n    Match sidekiq\n    Record fb.input tail\n\n[FILTER]\n    Name  record_modifier\n    Match puma\n    Record fb.input tail\n\n[FILTER]\n    Name  record_modifier\n    Match nginx-access\n    Record fb.input tail\n    Record logtype nginx\n\n[FILTER]\n    Name  record_modifier\n    Match nginx-errors\n    Record fb.input tail\n    Record logtype nginx-error\n\n[FILTER]\n    Name  record_modifier\n    Match rails\n    Record fb.input tail\n\n[FILTER]\n    Name  record_modifier\n    Match sidekiq\n    Record fb.input tail\n\n[FILTER]\n    Name  record_modifier\n    Match puma\n    Record fb.input tail\n\n[FILTER]\n    Name  record_modifier\n    Match nginx-access\n    Record fb.input tail\n    Record logtype nginx\n\n[FILTER]\n    Name  record_modifier\n    Match nginx-errors\n    Record fb.input tail\n    Record logtype nginx-error\n\n[FILTER]\n    Name  record_modifier\n    Match *\n    Record entity.guid.INFRA <REDACTED>\n    Record hostname ip-<REDACTED>\n    Record plugin.type nri-agent\n\n[OUTPUT]\n    Name                newrelic\n    Match               *\n    licenseKey          <REDACTED>\n    validateProxyCerts  false\n" file=/tmp/nr_fb_config681165494 process=log-forwarder
time="2020-11-01T06:09:08-06:00" level=debug msg="Launching process." component=integrations.Supervisor process=log-forwarder
time="2020-11-01T06:09:08-06:00" level=debug msg="Running command." command=/var/db/newrelic-infra/newrelic-integrations/logging/fluent-bit component=integrations.Executor env="[TERM=linux PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin PWD=<HIDDEN> SHLVL=0 UPSTART_INSTANCE= UPSTART_JOB=newrelic-infra DBUS_SYSTEM_BUS_ADDRESS=unix:path=/var/run/dbus/system_bus_socket]" path=/var/db/newrelic-infra/newrelic-integrations/logging/fluent-bit
time="2020-11-01T06:09:08-06:00" level=debug msg="/var/db/newrelic-infra/newrelic-integrations/logging/fluent-bit: error while loading shared libraries: libsystemd.so.0: cannot open shared object file: No such file or directory" component=integrations.Supervisor output=stderr process=log-forwarder
time="2020-11-01T06:09:08-06:00" level=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 127" process=log-forwarder
time="2020-11-01T06:09:08-06:00" level=debug msg="Supervisor backOff." backOff duration=3m25.269035876s component=integrations.Supervisor process=log-forwarder
time="2020-11-01T06:09:08-06:00" level=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender key=6380746073175715511 numEvents=2 postCount=25 timestamps="[2020-11-01 06:09:08 -0600 CST]"

@appsbeyond Been awhile since you posted this topic and unfortunately there hasn’t been a response from the community yet. I will raise this back up to our support team so that you can hopefully can get some help soon. Thanks for your patience!

Joi

@appsbeyond

Can you clarify what operating system and version you are using for EBS? Please also let us know what version of Infrastructure you are using.

From the looks of this, log forwarding only works on Amazon Linux 2 as the NR log forwarder is basically a wrapper around fluent-bit. This is a limitation from your underlying fluent-bit implementation. Unfortunately, can’t migrate to AL2 (which is relatively new) at this moment. I bet there are millions of deploys still on AL(1). Addressing the issue in fluent-bit will get this working for us all on legacy AL. Or creating a different implementation for Amazon Linux.

@appsbeyond Following up to see if you are still needing support from @rstarnes? From your last comment it looks like you have come to a conclusion about what would help you all get things working with your legacy AL.

Any update on this? Running into the same issue…

@JoiConverse / @rstarnes the last sentence by appsbeyond means that NR has to address the fluent-bit issue or create a different implementation of the log forwarding for legacy AL…

Legacy AL will be supported by Amazon for the next 3 years: https://aws.amazon.com/blogs/aws/update-on-amazon-linux-ami-end-of-life/ -> So it would definitely be great to have it supported by NR…

New Relic edit

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.

Hi @severin, I appreciate the feedback on this matter. I have submitted a feature request to our product development team to review. Hopefully this will be implemented further down the line and if so, I will update the thread accordingly.