Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store Β Β  Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Relic Solution: APM with Docker, Saving agent logs outside of your containers

docker
logging
apm
java
log
levelup
springboot

#1

Intro


So a quick intro: My name is Joseph and I am a Technical Support Engineer at New Relic. I mostly specialise in supporting our Java APM agent customers.

Normally when using Docker to containerise your application any log files generated by our APM agents are normally stored inside the container.

If you want to read these logs you normally need to use the docker exec command to connect to a running container and then try to read the logs via a text editor on the command line.

This is often not an ideal situation and it would be much more useful to be able to produce the logs on a local file system where you can read them comfortably in a text editor of choice, even if the docker container is not currently running.

If this sounds like something that interests you then you’ve come to the right place!

TL;DR


We can achieve this by making use of volumes and bind mounts in Docker.

https://docs.docker.com/storage/bind-mounts/

You can add the following lines to your docker-compose.yml file:

volumes:
      - type: bind
        source: ./logs/java-spring-app # The path to where you want the logs stored on your local machine.
        target: /application/newrelic/logs # The path to the logs folder inside the container.

It’s really as simple as that!

You can apply this to any of our APM agents and you should get the same results.

Java APM Example with Spring Boot


For this example I will use a Java application that is monitored by the New Relic Java APM agent.

Of course you should be able to adapt this method to any of our APM agents! (.NET, Python, PHP, Ruby etc…)

The Java application I will use for this example is built using the Spring Boot web application framework.

https://spring.io/projects/spring-boot

When booting up the Java agent alongside a Spring Boot application you must pass the -javaagent option as below:

java -javaagent:/full/path/to/newrelic.jar -jar app.jar

Your Java application may require different instructions for passing the -javaagent option to your JVM.

If your application is not using Spring Boot then please see our official documentation below for details on how to pass the -javaagent option with your desired framework:

https://docs.newrelic.com/docs/agents/java-agent/installation/include-java-agent-jvm-argument#spring-boot

I will also be using an OpenJDK 8 Alpine based image for my container.

https://hub.docker.com/_/openjdk/

This is just for example purposes, feel free to substitute this with your Docker image of choice.

Ok now that we have the prerequisites out of the way, let’s get down to business!

I have provided below a simple boiler-plate Dockerfile below:

Dockerfile

# Build an Open JDK 8 image based on Alpine Linux
FROM openjdk:8-alpine

# Copy "myApp.jar" file from inside the "/jars" directory. 
# The "/jars" directory is a relative path and should be at the same level as your Dockerfile (this file).
# Copy "myApp.jar" and place the new copy in the "/application/" directory inside the container.
COPY /jars/myApp.jar /application/

# Download the current version of the Java agent with wget
RUN wget https://download.newrelic.com/newrelic/java-agent/newrelic-agent/current/newrelic-java.zip

# Unzip contents into /application directory. Should now be '/application/newrelic'
RUN unzip newrelic-java.zip -d /application

ENTRYPOINT ["java"]

# Here I am passing the "-javaagent" option to the JVM
# We also pass the path that now exists inside the container to the "newrelic.jar" file.
CMD [ \
    "-javaagent:/application/newrelic/newrelic.jar", \
    "-jar", \
    "/application/myApp.jar" \
]

Above you might notice that I am downloading the newrelic.zip file from a URL that will always point to the most recent version of the Java agent.

https://download.newrelic.com/newrelic/java-agent/newrelic-agent/current/newrelic-java.zip

Please be aware that in the example Dockerfile above I am downloading the latest version for convenience.

Be aware that the config file contained in the downloaded file is missing a license key and so the agent will not boot up unless you explicitly pass a license key upon startup.

You can of course replace this step if you already have downloaded the Java agent and simply copy your existing newrelic folder into the container using the COPY syntax in your Dockerfile.

File Structure of newrelic.zip


This zip file contains a single folder called β€œnewrelic”.

Below is the file structure of the newly unzipped newrelic folder.

app/newrelic
β”œβ”€β”€ CHANGELOG
β”œβ”€β”€ LICENSE
β”œβ”€β”€ README.txt
β”œβ”€β”€ extension-example.xml
β”œβ”€β”€ extension.xsd
β”œβ”€β”€ logs
β”‚   └── newrelic_agent.log
β”œβ”€β”€ newrelic-api-javadoc.jar
β”œβ”€β”€ newrelic-api-sources.jar
β”œβ”€β”€ newrelic-api.jar
β”œβ”€β”€ newrelic.jar
β”œβ”€β”€ newrelic.yml
β”œβ”€β”€ nrcerts
β”œβ”€β”€ nrdiag
β”œβ”€β”€ nrdiag-filelist.txt
β”œβ”€β”€ nrdiag-output.json
└── nrdiag-output.zip

Or for those of your who are more visually oriented a screenshot showing the same from my text editor.

Please note that the logs folder will not be created until you have run the Java agent with your application at least once. This will create the logs folder and also the newrelic_agent.log file.

newrelic_agent.log is simply the default file name provided by the agent. This can be changed to your liking by editing your newrelic.yml file.

So the logs folder and the newrelic_agent.log file are really the only things we care about from above.

Docker Compose Section


So now we have a working Dockerfile that we can spin up your application along with the Java agent. Of course without a license key you will get an error message as explained above and the agent will not boot up.

So next we move to our docker-compose.yml file.

docker-compose.yml

version: '3.2' # This version must be 3.2+ for the correct bind mounting syntax

services:
  java-spring-app:
    build: ./app
    image: java-spring-app
    environment:
      - NEW_RELIC_LICENSE_KEY=your_license_key # Here you can optionally set the license key
    ports:
      - "8080:8080"
    volumes:
      - type: bind
        source: ./logs/java-spring-app # The path to where you want the logs stored on your local machine.
        target: /application/newrelic/logs # The path to the logs folder inside the container.
    tty: true
    stdin_open: true
    restart: always

The docker-compose.yml file should be one level up, on the same level as your /app folder in this example.

You can optionally feed in your license key to the container using the NEW_RELIC_LICENSE_KEY environment variable.

https://docs.newrelic.com/docs/agents/java-agent/configuration/java-agent-configuration-config-file#ev-NEW_RELIC_LICENSE_KEY

Running The Application


Now simply using the command docker-compose up on the command line your application should now boot up along side your new relic Java agent.

If a logs folder did not exist previously it will be created automatically along with the java-spring-app sub folder.

If you go here on your local file system you should now see the newrelic_agent.log file appear inside of the java-spring-app sub folder.

If you now bring down your container with the docker-compose down command you will see that the newrelic_agent.log file is still present. You can now open this with a text editor of choice and read away in comfort!

Conclusion And Additional Info


Now you may be asking: β€œWhy did you create a sub folder inside the logs folder?”

A good question! The reason is that docker bind mounts have no file locking in place, meaning that if you have multiple apps trying to write to the same folder you can get a conflict and corrupted data as a result.

It is recommended that you namespace your applications inside the logs folder, in this case I have used the name of the service but you could potentially choose any name you want here.

So as a final parting gift the important lines from above to make this magic happen are as follows:

volumes:
      - type: bind
        source: ./logs/java-spring-app # The path to where you want the logs stored on your local machine.
        target: /application/newrelic/logs # The path to the logs folder inside the container.

It’s really as simple as that!

You can apply this to any of our APM agents and you should get the same results as above.

Just make sure you know the path that your agent is using inside the container to store logs.

And with that congratulations! You now know how to save your agent logs locally outside of your container.

I hope this has been a helpful example!


#2

Hi there,

thanks for this nice example to get the logs via docker-mounts.

We also thought about doing this in our micro service-setup on aws, but then went another route to get the logs out of our containers.

As we allready used filebeat on the docker-host, we updated to the V6 which is capable of getting the logs of each container and ship it to our logstash-server:
β€œhttps://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-docker.html”

It was lightweight for us - just another way to do the same thing :wink:

hartmut


#3

That’s another great solution @hartmut.koenig - Thanks for sharing.