Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Relic Solution: What you need to know about New Relic, when deploying with Docker

deployment
nodejs
docker
apm
java

#1

Intro

(Before we get started…did you know you can toggle the new OTC look and feel between “New Relic Light Theme” and “New Relic Dark Theme”? Pull down the hamburger menu in the upper right, and scroll it down to see the controls.)

OK.

Let’s cut to the chase right away about this post: it isn’t going to be a direct answer to “tell me specifically how to write my Docker and Docker Compose files, if I’m deploying New Relic.”

We will be showing some general Docker examples - in fact we’ll show you a complete working example with of a dockerfile and a docker-compose file, in just a second. But the main point of this post is to show you the key concepts you need to know about New Relic, if you are using Docker. From there, we’ll go over some general Docker concepts you can use - for example, using Docker to set environment variable values inside your Docker containers.

We’re going to be using New Relic’s APM agent for Java as our example, throughout most of this doc. But the learnings you’ll get will easily apply to our other language agents - also, to our New Relic Infrastructure agent - as we’ll show towards the end of the post.

One other note: a lot of the learnings in this post will be relevant beyond Docker examples. For instance, if you are writing up your own deployment scripting, or building out custom Puppet, Chef, Ansible, Salt, or really any other CM scripting, there will be quite a bit of relevant information in this post to help guide you.


Sections of this post

Let’s say that you were testing out deploying a single Java web application, using Docker and New Relic. These 2 files might be all you need:

$ cat Dockerfile

FROM tomcat
RUN mkdir -p /usr/local/tomcat/newrelic/logs
RUN useradd tomcat
RUN chown -R tomcat /usr/local/tomcat/newrelic/logs
ADD ./newrelic/newrelic.jar /usr/local/tomcat/newrelic/newrelic.jar
ADD ./newrelic/newrelic.yml /usr/local/tomcat/newrelic/newrelic.yml

$ cat docker-compose.yml (you’ll need to correct the indents on this one if you want to try it out yourself…)

version: ‘3’
services:
web:
build: .
ports:
- “8080:8080”
environment:
- TOMCAT_USERNAME=tomcat
- TOMCAT_PASSWORD=tomcat
- JAVA_OPTS=-javaagent:/usr/local/tomcat/newrelic/newrelic.jar

Section 1 of this post will explain the lines above which are specific to New Relic, and why you need to be thinking about each one of them.

Section 2 of this post will be about configuration of a larger environment or set of environments, perhaps orchestrated by Kubernetes, in which you need to be thinking about the bigger picture of using Docker to configure New Relic, in an efficient way.

For example, let’s say this is more representative of your environment:

In section 2, we’re going to assume that you’re comfortable with doing things like:

Using Docker to set environment variables inside Docker containers: https://docs.docker.com/compose/environment-variables/

Using Docker to set java system properties inside Docker containers: https://www.google.com/search?q=docker+set+java+system+property

Why? Because New Relic’s agent config can have its values either set, or overridden, using solutions such as Java system or OS environment variables. We’ll walk through examples of these configuration mechanisms, in section 2 of this post.

[UPDATE: see the comments section below: all Java agent configuration settings can now be set with Env vars - removing the need for using system properties.]


Section 1: Deploying a single application server with New Relic

When you first start using New Relic, one of the first things we show you is how to quickly install our agents from the command line. Take Java as an example: our UI shows you the easy steps you need to follow, and in a few minutes you’ll have the agent up and running:

(That’s after downloading the correct agent zip file onto the host in question.)

When you are trying out New Relic for the first time, perhaps on a laptop or some server you have access to, that’s a great way to see how quickly you can install New Relic’s APM agent, and start seeing your performance data in just a few minutes.


If you are deploying an agent for the first time: I highly recommend you let the installer do it first, and then take a look at what it did for you. For example, follow the in-UI directions to have it unzip to /usr/local/tomcat/newrelic (obviously that assumes Tomcat - you’ll do the right thing for other web app servers you’re running), and then go look at what the installer did to your catalina.sh file (again, that’s using Tomcat as our example). As of this writing, the latest version of the agent added the following to my catalina.sh file for me:

NR_JAR=/usr/local/tomcat/newrelic/newrelic.jar; export NR_JAR
CATALINA_OPTS="$CATALINA_OPTS -javaagent:$NR_JAR"; export CATALINA_OPTS

(As you’ll see In just a bit, I’ll go into more details about that “-javaagent” argument…)

If you let the installer do this setup work for you the first time, it will give you a great example of what you’ll next start to do manually, as you start building out New Relic deployment with Docker.

Let’s take a look at some key concepts you’ll want to think about, as you start assembling the Docker commands you need.

1) Where does the agent live on the host?

In our documentation, we typically recommend that you install our agent somewhere relative to your application server. For example, if Tomcat is deployed at /usr/local/tomcat, then we recommend that you unzip the agent jar file to /usr/local/tomcat/newrelic. However, this is just a recommendation. You could just as easily unzip it to /usr/local/tomcat/newrelic, or /teams/myteam/myproject/newrelic, or really any directory that makes sense for your project and your organization.

A very important thing to know: your startup script (the one which contains the command that starts your application server) must include Java’s built-in argument called “-javaagent”, and the value of that argument must be the location of wherever you unzipped the newrelic agent. For example, “-javaagent:/usr/local/tomcat/newrelic/newrelic.jar”.

We detail a bunch of ways to do this with startup scripts:
https://docs.newrelic.com/docs/agents/java-agent/installation/include-java-agent-jvm-argument

For example:

In our Docker example files above, we had a couple of lines specific to this concept:

1: In the dockerfile:

ADD ./newrelic/newrelic.jar /usr/local/tomcat/newrelic/newrelic.jar

2: In docker-compose.yml:

environment:
  - JAVA_OPTS=-javaagent:/usr/local/tomcat/newrelic/newrelic.jar

We’ve loaded our newrelic.jar into our /usr/local/tomcat/newrelic directory, and then set the -javaagent flag using the “environment” stanza of our docker-compose file. As you saw in that screenshot from our docs, there are many ways you can set the -javaagent flag with your startup scripts; using docker-compose is just another way.

Now let’s move on to the agent’s configuration file, called “newrelic.yml”.

2) Where does the agent’s configuration file live on the host?

For the Java APM agent, the config file is called “newrelic.yml”, and It typically just lives in the same directory as the agent itself.

Alternately, the config file can be in some other location of your choice, and in this case you would use an environment variable to tell the agent where its config file is. See this page for details if you choose to place it somewhere else:
https://docs.newrelic.com/docs/agents/java-agent/configuration/java-agent-configuration-config-file

For this article, let’s just assume you are placing the newrelic.yml file into the same directory as newrelic.jar.

Once again, looking at our Docker example above, let’s see how we managed this:

ADD ./newrelic/newrelic.yml /usr/local/tomcat/newrelic/newrelic.yml

3) Where does the agent write its log files?

By default, into a ./logs directory, relative to the location of the newrelic.jar file. Again, this is something you can control, using settings.

MAKE SURE that the user account which starts your application server also has the right to perform any tasks such as creating the logs directory, and creating and appending to the log files in there. This requirement gets missed sometimes, and then the agent can’t create its logs.

Here’s how we did that with Docker:

RUN mkdir -p /usr/local/tomcat/newrelic/logs
RUN useradd tomcat
RUN chown -R tomcat:tomcat /usr/local/tomcat/newrelic/logs

Again, this is totally up to you to build the way you need it. Our example assumes that the “tomcat” user is responsible for starting Tomcat - this obviously could be a different user in your case.

4) What else?

If you restart your server and the app doesn’t show up in New Relic in the first few minutes, what do you need to re-check?

  • -javaagent flag correctly points to the agent?
  • license key and app name set into config file?
  • owning process can create logs/ dir and write files there?
  • can your agent call outbound through your firewall?

For that last one, you might want to review these 2 links in our docs:

https://docs.newrelic.com/docs/accounts-partnerships/accounts/security/data-security#data_transmission

https://docs.newrelic.com/docs/apm/new-relic-apm/getting-started/networks

If things still aren’t working: try using your config file to set log level to FINEST for a few minutes, restart the app server, and see what you see in the logs.


So, concluding section 1, we’ve seen how easy it is with a short Dockerfile and docker-compose.yml, to spin up a Tomcat instance that has New Relic installed. If you were to try this out, you would first need to download newrelic.jar and newrelic.yml into the local “./newrelic” directory we specified in these lines:

ADD ./newrelic/newrelic.jar /usr/local/tomcat/newrelic/newrelic.jar
ADD ./newrelic/newrelic.yml /usr/local/tomcat/newrelic/newrelic.yml

And you would need to go into newrelic.yml and at the very least, put in your license key and a reasonable application name, in the appropriate places. Section 2 will give many more details about these and other configuration settings.

We’ll also see in section 2 how you could control these (and other) configuration settings on a larger scale, as opposed to going in and editing each one of your configuration files individually across your containerized environments.


Section 2: Deploying many application servers with New Relic

Let’s go back to that visual example for a moment:

Take those 3 nodes at the bottom. Let’s say those represent just 3 of your environments - for a simple example, dev, stage, and prod. In each one, you are probably going to deploy a cluster of docker containers. Many of those containers will need New Relic APM agents on them.


[ And now, for a word from our sponsor… ]

All of your nodes should also have the New Relic Infrastructure agent on them!

https://docs.newrelic.com/docs/infrastructure

[ Ok, back to our program… ]


Here’s some questions you might already be asking:

  • Can I deploy New Relic in a way that I can have 100% identical configuration files for each of my containers, regardless of what environment my containers are going to land in?

  • If every agent in every environment needs slightly different configuration data, how can I control that all, from the Docker layer?

  • What if I want the New Relic agent to be disabled by default in some environments, and only enabled in others which I specify?

In section 1, we saw what it might be like to set up New Relic APM for a web application server (like Tomcat, or the many other app servers we support). As we get further into talking about Docker on a larger scale, let’s look more closely at some different methods by which the agent can pick up its configuration information at boot time, besides just reading its own local configuration file.

As we mentioned earlier, the New Relic Java agent’s config settings can be set (or overridden) in a few additional ways:

  • By Java system properties,
  • By OS-level environment variables.

[UPDATE: see the comments section below: all Java agent configuration settings can now be set with Env vars - removing the need for using system properties.]

For a visual of how this works in terms of precedence ordering, go here: https://docs.newrelic.com/docs/agents/java-agent/configuration/java-agent-configuration-config-file#config-options-precedence

(That diagram shows that you can even override all other configuration methods, by using “server-side settings” - i.e. directly from the New Relic UI. This topic won’t be covered here, but it’s at least good to be aware of.)

As we move deeper into this section of Java agent configuration, there’s an important detail to notice in our docs. The section on Java system properties:
https://docs.newrelic.com/docs/agents/java-agent/configuration/java-agent-configuration-config-file#System_Properties

mentions that “you can override any setting in the newrelic.yml file by setting a system property.”

However, notice that the next section about environment variables:
https://docs.newrelic.com/docs/agents/java-agent/configuration/java-agent-configuration-config-file#Environment_Variables

lists just 4 configuration settings which can be set/overridden via this method:

  • NEW_RELIC_APP_NAME
  • NEW_RELIC_PROCESS_HOST_DISPLAY_NAME
  • NEW_RELIC_LICENSE_KEY
  • NEW_RELIC_LOG

These 4, especially app_name and license_key, may be the only configuration settings you need to change, to get started with your deployment. However, if you need to go deeper into controlling other settings via Docker, you’ll need to be thinking in terms of Java system properties for those. (We’ll look at the same concepts for a different APM agent (Node), towards the end of this post.)

The main thing to keep in mind: by using solutions like environment vars and system props, you can control your New Relic configuration settings completely from the Docker level.


Let’s look a little more deeply into some of the configuration settings you’ll want to consider as you move into dockerizing your New Relic deployments. First, we’ll ask the question: “What New Relic config settings absolutely have to be provided, before I start up my application server?

  1. Required: the agent config must contain your license key, so that the agent can write data into your account.

  2. Highly recommended: you should also update the config’s “application name” setting, so that instead of the default “My Application”, it says something specific about the environment, and about the role of the application; such as, “DEV web front end”.

Whatever the value is that you put into this setting, that is what you are going to see for the name of the application server, when you go into the New Relic UI.

Side note: you can re-use an application name for multiple apps serving the same role - so that all the data from those apps rolls up into the same “logical” application in our UI. For more detail about additional grouping options, look here: https://docs.newrelic.com/docs/agents/manage-apm-agents/app-naming/use-multiple-names-app

  1. Also recommended (especially as we get towards the Docker conversation): in the newrelic.yml file, scroll to the end, and look at the “environments” section. What this is saying is that if you don’t specify a value for “newrelic.environment” (via a Java system property), the agent will assume that it is running in your production environment, and simply use all of the values from the main body of the config file.

But, if you do use a java system property to specify your environment, such as “newrelic.environment=dev”, then the agent will use specific settings from the “dev” stanza at the bottom of the config file (plus all the rest of the “default” settings in the main body of the file.) In our out-of-the-box newrelic.yml files, you can see that we include simple examples for naming your application, specific to some environments it might be deployed into.

So, already you can see that by using java system properties (and/or OS-level environment variables), you can do a couple of key things:

  1. Tell the New Relic agent which environment it’s deployed into, and
  2. Control all of the configuration settings for the agent.

Seems like a good way to be thinking, as you are looking ahead at deploying with Docker, right?

Let’s take an example.

  • I have a Docker container which will contain my web front-end (let’s say Tomcat)
  • I plan to deploy it in many environments (dev 1/2/3, QA, staging, production, etc).
  • I want my container to also have New Relic installed (say, at /usr/local/tomcat/newrelic).
  • I don’t want to have container-specific New Relic config files, for each container deployment, and for each environment that container might be going into.
  • I need the default setting to be that the agent is disabled, and using environment vars, I want to be able to control that an agent becomes enabled, in containers which I specify at the Docker level.

What configuration settings should I be thinking about setting up?

1) newrelic.environment
If you choose to use this, it needs to be set via a Java system property - it’s not available as an environment variable.

The environment setting will control which of the environment-specific stanzas the agent should pull its config settings from.

Why would you want to do this? You might consider this choice: whether you want to code all of your environment-specific configuration settings within your newrelic.yml files, or whether you want to set them via Docker. So your choices here might be:

  1. If you want to just have a completely identical newrelic.yml in every container in every environment, and then use Docker to pass in all of your environment-specific config values via sys props or environment vars, then you could just skip using the “newrelic.environment” setting. Every environment-specific config value will come directly from Docker, and you could just leave newrelic.yml in its out-of-the-box state.

  2. You could control all of your environment-specific settings within the environment stanzas at the bottom of newrelic.yml. Then, you would simply pass in the “newrelic.environment” Java system property via Docker, to tell New Relic which of your environment stanzas to use.

Totally your choice - we offer you the flexibility to design your system your way.

So let’s see an example of passing in “newrelic.environment”. Taking our example above of Googling for “java set system property”, let’s look at this first result from https://stackoverflow.com/questions/33408626/how-to-pass-system-property-to-docker-containers

docker run -it -e “ENV=dev” myDockerImage
java -Denvironment=$ENV -jar myjar.jar

That’s just one example. We could tweak that second line just a bit, to:

-Dnewrelic.environment=$ENV

Again, in this post we’re not going to try to give you exact instructions for using Docker - but this might be one way you could pass a system property to Java, which the New Relic agent would then pick up and use. (You can also see a different way of passing in JAVA_OPTS, in the initial docker-compose file at the top of this post.)

2) license key
You could control this either with a system property or an environment variable.

As mentioned, this field must be set in some way, or else you won’t report any data into your New Relic Account.

3) application name

You could also control this either with a system property or an environment variable.

By the way, this one’s super easy to control from the Dockerfile. In our example at the top of this post, you can just add this line:

ENV NEW_RELIC_APP_NAME=“My App Name from a Dockerized Env Var”

and you should see that as your application name in New Relic, next time you boot the container.

Here’s some examples of application naming you might use:

DEV_1_web_front_end
DEV_1_middle_tier
DEV_2_web_front_end
DEV_2_middle_tier
PROD_web_front_end
PROD_middle_tier

(As you might have guessed by now: that example would certainly work for setting your New Relic license key too)

ENV NEW_RELIC_LICENSE_KEY=“My License Key from a Dockerized Env Var”

4) agent enabled/disabled

Let’s look at an example. You have a docker image, and you want to use the same image for every install. However, you don’t want to run the New Relic agent every time an engineer spins up a test app, because you don’t want to run up your instance count. If you don’t know ahead of time which instances will need the agent installed, how do you choose whether or not to enable the agent at runtime?

This problem can also be solved using the newrelic.environment flag. You can disable the Java agent by setting “enabled: false” in the main body of newrelic.yml file, but set then set “enabled: true” in specific environment stanzas, and voila! Now you can run the agent or not by specifying the environment at runtime.


One more question: can I install New Relic via a Docker volume?

Answer: Certainly, you could install New Relic into a Docker volume, and then make it available to your app. This might in fact be the most flexible and least intrusive solution for your project.

For this post, we’ll leave that as a question for you to decide, and also for you to choose how to implement with Docker.


Conclusion

With the above examples, you should be a in a good position to start configuring New Relic via Docker, for simple examples like in section 1, or for more complex situations like in section 2.

Also, if you’re reading this with other solutions (for example, config management tools like Puppet/Chef/Ansible/Salt), perhaps this post has been helpful for you to plan out your strategies and implementations with them.

What about the other languages, for which we offer APM agents? We’ll, let’s start by showing you the first place you want to go: https://docs.newrelic.com/

Let’s say you’re working with the Node APM agent. So, clicking the “Node.js agent” link takes you here: https://docs.newrelic.com/docs/agents/nodejs-agent

You’ll then want to click the “Install the Node.js agent” link, to get here:
https://docs.newrelic.com/docs/agents/nodejs-agent/installation-configuration/install-nodejs-agent

From there, you’ve got the same information we presented above for Java:

  • Steps to manually install the agent,
  • Steps to modify the agent’s configuration file.

Configuration via channels (env vars, sys props, etc.) is one area where you’ll see slight differences between our language agents. Check out the Node agent documentation here:
https://docs.newrelic.com/docs/agents/nodejs-agent/installation-configuration/nodejs-agent-configuration#environment

For example:

So as you get to know your available choices for any agent, the next question will be: “how can I control my configuration settings via Docker?” Once again, that’s going to be something you’re going to solve directly via your own usage of Docker, but I hope that the information presented here will be all that you need to understand the New Relic end of things, and perhaps even to get some new ideas for the Docker end.

Finally, while you’re working on your install and configuration of the APM agent, you’ll probably find that installing the Infrastructure agent is a breeze. Here’s the relevant documentation for that agent:

https://docs.newrelic.com/docs/infrastructure/new-relic-infrastructure/installation

Note that for New Relic Infrastructure, we also provide ready-to-go deployment examples for Puppet, Chef and Ansible, in the docs:

https://docs.newrelic.com/docs/infrastructure/new-relic-infrastructure/config-management-tools

Happy Containering!!!


Relic Solution: Installing Java agent in AWS Elastic Beanstalk Application
Newrelic not running on Openshift when running as INIT container
#2

Update: Beginning with version 4.10.0, “The Java agent now supports setting all configuration options via Environment variables.” It is no longer necessary to use system properties for configuration settings.


#3

That’s a great thing to point out, thanks @philweber. This whole article could probably use some rewriting at some point, including easy examples of using Docker Volumes to include the agent. In fact, here’s a very simple example, where I started from the sample project I will link below, and make just a few changes to include NR APM like this (no Dockerfile needed, just using docker-compose.yml)

version: ‘3.2’
services:
petclinic:
image: tomcat
ports:
- “8000:8080”
volumes:
- type: bind
source: ./newrelic
target: /usr/local/tomcat/newrelic
environment:
- “JAVA_OPTS=-javaagent:/usr/local/tomcat/newrelic/newrelic.jar”

This example could easily be used to also demo any kind of Env var, like you suggested.