New APM Agents now include automatic log configuration, log metrics, and log forwarding built directly into the agents

Proven Practices Level Up Relic Solutions
,
#1

New Relic released an important new capability for APM language agents that integrates application logs with APM. When you upgrade to the latest versions of the Ruby, Java, and .NET agents, the APM agent will add entity context and forward the logs directly to New Relic, eliminating the need for third-party applications to be installed or configured.

Starting May 3rd, 2022, you get the following three key benefits when you update to these new APM agents:

  • Eliminate manual log configuration and maintenance necessary to collect and forward logs to New Relic when using Java, .NET, and Ruby APM agents.
  • Minimize context-switching by viewing application logs in context with APM metrics, traces, and events.
  • Troubleshoot issues faster by accessing logs inside the APM with enhanced UI that help you troubleshoot issues faster.

When collecting logs, it is critical to ensure security, compliance, and control. This is why we are including robust support for the following:

  • Opt-out anytime: Turn off automatic forwarding at any point by configuring your agent or using New Relic’s data management hub.
  • Ingest control: Use in-agent log sampling to manage ingested volume and avoid duplicating ingest and get 3X more value than alternate log management solutions.
  • Compliance: Log collection is disabled by default for HIPAA, PCI-enabled accounts, and accounts where High Security Mode is in use, even after you upgrade the agent.
  • Data security: Mask, obfuscate, and prevent sending PII, PHI, or any other sensitive data via customizable security configurations.

Note: Application logs in APM is only available after an APM agent update. Until you upgrade, you will see no changes in your account.

APM Agents with logs in context

Logs are a critical telemetry type for observability and now, with automatic logs in context available for APM agents, you can immediately view relevant logs associated with other application telemetry data so you can find and fix issues faster.

Currently supported APM languages and frameworks:

  • Java
    • Log4j2
    • Logback
  • .NET
    • Log4net
    • Serilog
    • Microsoft.Extensions.Logging
  • Ruby standard library logger

What you’ll need

Before you begin, ensure you have a New Relic account, or sign up for a free account here (no credit card needed).

You will also need to have the latest version of the Java, Ruby, and .NET agents:

Avoid duplicating log data

In certain scenarios, you might end up with “duplicate” logs being collected. This could be because the same log information is being sent to New Relic twice (for example, if you use a third-party log forwarder.) To avoid duplicating log data, consult this documentation.

What about sensitive log data being sent to New Relic

You control what log data is sent to New Relic, and collecting logs using the APM agent is no different. Be sure to follow your organization’s security guidelines to mask, obfuscate, or prevent sending PII, PHI, or any other sensitive data. To learn more about New Relic’s log management security and privacy click here.

Please note that no logs of any kind will be collected if High Security Mode is enabled on the agent, even after the agent is upgraded. It is also possible to configure drop filters to help prevent sensitive data from being stored in New Relic.

Turning automatic log forwarding off

The feature will be on by default for new installations or upgrades starting May 3, 2022. If you want to halt logs ingest:

  • You can configure the agent config file locally on a machine to disable it.
  • You can disable logs ingest for APM agents at the account level with a toggle in the New Relic data management hub1. The toggle can be flipped before ever setting up an APM agent that forwards log data. Shutting off the logs at the account level is an all or nothing choice. There is no way to allow-list specific agents to send data via controls within the New Relic UI. Disabling logs via UI may cause a higher-level of resource utilization. Please disable logs via the agent config file if you have resource-sensitive systems.

Support

For more information about obfuscating sensitive data in logs, read these docs or contact Support.

3 Likes
New Relic Node.js and Python APM agents to include automatic logs in context starting July 21st, 2022!
Partitioning logs from multiple distinct Heroku apps
#2

Does this mean that this outstanding issue will be rendered moot [edit: for supported APM agents]? Will this Just Work, then?

#3

How does the automatic logging deal with multiline output such as java stacktraces? Will automatic logging put all those lines into one message or is each line of the stacktrace ingested as a separate message?

#4

Is there a way to filter out certain log event “levels” to prevent them from being shipped using the built-in log forwarder? For example, we have some log entries made at Debug level that we store locally, but want to ship only Error and Fatal messages up to NewRelic. The advantage of log4net and similar is that we can configure how to handle each level independently of our code. I don’t see anything in the documentation about pre-limiting messages by level.

#5

Okay, I’m going to save others a little panic. It’s not obvious from the email (which definitely ought to have come much, much sooner), but there’s a page buried in one of the links above Disable automatic logging | New Relic Documentation that shows how to disable this abrupt shift in default.

NewRelic, seriously, it’s great that’s an option now, but to spring it on everybody 2 business days ahead of time?

4 Likes
#6

Thanks for your feedback, I’ve added some clarity to the post.

#7

Agreeing with ntkach here, and not finding much clarity in the post above.

We caught this just in time to disable the automatic logging, others may not be so lucky.

#1. We do NOT wish to have data replicated to New Relic unless we choose to do so. Especially application log data with who knows what potentially sensitive information.

#2. We are NOT amused to know that this “feature” would be activated if at any point we should install an update to an APM agent. We happen to be in the middle of a server migration, and could easily be installing the latest version of an APM agent.

#3. Data ingestion could well go through the roof, potentially raising the cost of using New Relic beyond what we budgeted for.

All in all this makes us a less happy camper.

3 Likes
#8

@ymireles - Java agent v7.7.0 has not yet been released. Does this change impact 7.6.0 agent version?. Please clarify

#9

This feature is available on the Java agent v7.6.0 or higher (post corrected)

Thank you

#10

it1298, we understand your concerns, with New Relic you are always in control on what data you send to us. Let me do my best to address your concerns here:

  • How is security handled? Is there a way to filter out certain log events

    • We automatically mask credit cards and social security numbers.
    • A new Obfuscation UI and customizable security configurations to prevent sending PII, PHI, or any other sensitive data have stream filtering and obfuscation controls (Data Plus).
    • Customers can control the sampling rate in their newrelic.yml config
    • New Relic has SOC2, ISO27001 and HITRUST.
    • Logs disabled for HIPAA, PCI-enabled accounts, and High Security Mode (even after upgrading)

  • You can Opt-out at anytime, even before updating the agents:

We are also seeing a low average daily log volume for ingestion when this is on, but you can reach out to your customer representative to find out more.

You can find more information here

#11

It think you’re missing a couple of points here. This was implemented as an opt-out feature, which is the wrong way around, and it was announced on April 28 as going into effect on May 3, which is way too short notice. There is also the issue that the feature may be deployed via an incidental update to the agent software, unless the end user was aware of the required opt-out before the fact.

Happy to hear the estimated average daily log volume for ingestion is low, but then again I wasn’t aware that our Infrastructure Agent is feeding enough uninteresting data about OS processes that it takes up 75% of the monthly free tier of data, for just a handful of servers. And there’s no simple opt-out in the account settings for that.

Still, overall mainly a happy customer.

2 Likes
#12

I’ve tried this feature out on our test system which is using the Java agent and Logback for logging. While it does work well it seems to have a few shortcomings. Noticeably:

  1. No ability to switch log level. It seems to completely ignore the logback.xml config and just log everything down to debug level. I read above that you can indeed ignore that data in New Relic however I would prefer to not send it at all due to it adding a network overhead etc.
  2. The class and line number logging by logback isn’t sent through to New Relic so I’m losing information. This is the biggest issue and because of this I’ll have to disable the feature and revert to capturing that data some other way.

It would be great of those things could be addressed and would make the service a lot more useful.

1 Like
#13

Exactly, while I don’t doubt this is something all sorts of people might want/use, the fact is New Relic has a lot of Enterprise users. We’re not free to just suddenly start sending anything that we happen to log at the time to NewRelic (regardless of how secure NewRelic might be handling it or might allow us to change after the fact).

If you (NewRelic) are going to make a change like this opt-out, then you need to give us a prominent (not just a casual thread on this site that we might know to check) warning well ahead of time (like at least 30 days out, ideally more like 90). I’m not sure NewRelic quite understand the sheer volume of logging that we might have enabled at any given time. For all you know when this thing flipped on we might have fully verbose debugging on. We could accidentally start sending hundreds of gigabytes an hour and not know it until we suddenly start getting bills/alarms.

Yep, exactly our experience with infra agent too.