Multiline log parsing with AWS FireLens

I have log forwarding setup using the New Relic Fluent Bit image for AWS FireLens and I need to configure the parsing options to concatenate multiline log messages. I am trying to extend the new relic image from 533243300146.dkr.ecr.us-east-2.amazonaws.com/newrelic/logging-firelens-fluentbit so I can pass in my own Fluent Bit config files, but I am unable to pull the image (error: no basic auth credentials).

Is there an official way to configure Fluent Bit parsers using the New Relic plugin image?

Hi @corey17,

Welcome to the community!

Here is the link to configuration examples that can help you check the details: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit
There is an example of multiline as well.

I hope it helps.

Thanks,
Suman

1 Like

Hi, thank you for the reply.

I am looking to configure the parser using the New Relic FireLens plugin images listed in the newrelic/README.md example you linked. I do not see how to set parser options here.

The other example for configuring multiline parser is building a custom fluent bit image with config override files. This would require building the new relic integration myself for this use case.

These two examples do not seem compatible. Is it possible to configure the new relic plugin images or do I need to build my own custom fluent bit image?

Hey @corey17,

This isn’t something that we support, but it may be possible.

You will need to use S3 to host both a fluent-bit.conf file and a parsers.conf file.

You could then pull those files into your task definition by adding this to the firelensConfiguration:

        "firelensConfiguration": {
          "type": "fluentbit",
          "options": {"config-file-type": "s3","config-file-value": "arn:aws:s3:::bucket-name/fluent.conf", "enable-ecs-log-metadata": "true"
          
               }
        }
      },

You could then try following this blog here to enable multiline parsing:

Configuring the [INPUT]section of the fluent-bit.conf file might take some nuance, as this isn’t something that we support for FireLens, but this should hopefully lead you down the right path.

Thanks,
DJ

Thanks, dzevenbergen!

I definitely missed that config file option in the docs for Firelens and I think that is exactly what I need.

Regards,
Corey

For those using Fargate, which does not support 'config-file-type': 's3', I’ve come up with a (somewhat awkward but effective) solution to provide a custom Fluent Bit config with the newrelic/logging-firelens-fluentbit image using 'config-file-type': 'file'.

Basically, you can override the command on the ECS task definition and write your config file to disk before starting the fluent-bit process:

      command: [
        'mkdir -p /var/etc/fluent-bit/ && ' +
        'touch /var/etc/fluent-bit/fluent-bit.conf && ' +
        'echo '[FILTER]' >> /var/etc/fluent-bit/fluent-bit.conf && ' +
        'echo '    Name record_modifier' >> /var/etc/fluent-bit/fluent-bit.conf && ' +
        'echo '    Match *' >> /var/etc/fluent-bit/fluent-bit.conf && ' +
        'echo '    Record environment ${ENVIRONMENT}' >> /var/etc/fluent-bit/fluent-bit.conf && ' +
        'exec /fluent-bit/bin/fluent-bit -e /fluent-bit/firehose.so -e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so -e /fluent-bit/bin/out_newrelic.so -c /fluent-bit/etc/fluent-bit.conf',
      ],
      environment: [
        {
          name: 'ENVIRONMENT',
          value: 'dev',
        },
      ],
      firelensConfiguration: {
        type: 'fluentbit',
        options: {
          'enable-ecs-log-metadata': 'true',
          'config-file-type': 'file',
          'config-file-value': '/var/etc/fluent-bit/fluent-bit.conf',
        },
      },
2 Likes

Thank you for this! It got me started down the path for condensing partial messages. Here’s where I ended up if it helps anyone else (typescript - that’s the reason for the escapes):

entryPoint: ["sh", "-c"],
command: [
  "mkdir -p /var/etc/fluent-bit/ && " +
    "touch /var/etc/fluent-bit/fluent-bit.conf && " +
    'printf "[FILTER]\\n    name record_modifier\\n    match *\\n    record sample-field ${ENVIRONMENT}\\n" >> /var/etc/fluent-bit/fluent-bit.conf && ' +
    'printf "[FILTER]\\n    name multiline\\n    match *\\n    multiline.key_content log\\n    mode partial_message\\n" >> /var/etc/fluent-bit/fluent-bit.conf && ' +
    "exec /fluent-bit/bin/fluent-bit -e /fluent-bit/firehose.so -e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so -e /fluent-bit/bin/out_newrelic.so -c /fluent-bit/etc/fluent-bit.conf",
],
1 Like