Relic Solution: Using custom SSL certificates for CPM communication with your endpoint

Introduction

Here we detail how to make custom images for the minion (ping jobs) and runner (non-ping jobs) that allow containers created from them to establish a SSL handshake with endpoints, defined in Synthetics monitors, that need to use your Certificate Authority or root certificate file in order to connect.

If there is an issue connecting to your endpoint due to a SSL handshake failure, you may see a monitor error like Generic network error: -2 and an error like verify error:num=20:unable to get local issuer certificate when testing with OpenSSL from inside the synthetics-minion container.

$ docker exec -it YOUR_MINION_CONTAINER bash
root@473c819fa555:/opt/newrelic/synthetics# echo | openssl s_client -connect host:port

A few different certificate related requirements in a private network can necessitate modifying the existing synthetics-minion and synthetics-minion-runner Docker images. These customization steps will need to be repeated any time there’s a new version of the minion container or if any additional certificate modifications are required. These solutions are provided for the benefit of our customers, but are not supported by New Relic Technical Support*.

The custom Docker images created during this process can be added to an image repository or exported / imported using docker image save -o <filename>.tar <image> and docker import <filename>.tar <image>.

Horde Communication

Private minions retrieve monitors to execute from the Horde. Typically certificate customizations are not required, but a well documented solution to customize the certificates used for communication between the Horde and your minion is available. See this excellent post by @Michel_L.

Monitors Using Verify SSL Configurations

The SSL verification portion of ping and simple browser monitors executes from the synthetics-minion container. Use these steps to build a custom synthetics-minion Docker image that includes additional certificates. These will allow you to execute ping and simple browser monitors with verify SSL configurations for applications with self-signed certificates, untrusted root certificates, etc.

  1. Download the certificates that need to be added to your Chrome configuration. This may include root, intermediate, and server certificates. Suggestions on how to download these certificates are included in a later section of this document. Name your certificate files well because each will need to be copied into the synthetics-minion container image at a later step. Make sure to save or rename each file to use a .crt file extension before proceeding.

  2. Copy all of the certificates that need to be added to the minion into a single directory on a system that has Docker installed.

  3. On the same system with Docker installed, create a file named Dockerfile with the following content, which will be used to build a custom synthetics-minion image. Make sure to update the COPY step to use the location that your certificates were saved. The example below assumes that they are saved in a subdirectory of the directory that contains your Dockerfile that is named certificates:

    FROM quay.io/newrelic/synthetics-minion:latest COPY ./certificates/ /usr/local/share/ca-certificates/
    RUN update-ca-certificates
    
  4. Build the custom Docker container from the Dockerfile:

    docker build --tag synthetics-minion-certificate .
    
  5. Start the custom synthetics-minion container using the docker run command provided on the Private Locations page and the version used in your Dockerfile and docker tag command, making sure to replace the private location key:

    docker run -e MINION_PRIVATE_LOCATION_KEY=abc123 -v /tmp:/tmp:rw -v /var/run/docker.sock:/var/run/docker.sock:rw -d --restart unless-stopped synthetics-minion-certificate:latest
    

    This command may vary if custom node modules, Verified Script Execution (VSE), or other optional settings are configured.

  6. Execute your monitor and verify it is successful.

Modify Certificates in Chrome

These steps can be used to add self-signed certificates, etc. to Chrome in the synthetics-minion-runner container so that sites will be trusted in scripted browser monitors. While Synthetics typically ignores certificate errors, there have recently been a few scenarios where untrusted certificates have caused monitors to fail during execution. Including additional certificates requires the use of a private minion and the customization of the synthetics-minion-runner container image.

  1. Download the certificates that need to be added to your Chrome configuration. This may include root, intermediate, and server certificates. Suggestions on how to download these certificates are included in a later section of this document. Name your certificate files well because each will need to be copied into the synthetics-minion-runner container image at a later step.

  2. On a system with Docker installed, create a file named Dockerfile in the same location as your certificates. This example file will need to be modified to include steps to copy each certificate as well as including multiple certutil commands to import the certificate for Chrome’s use. Make sure to edit both the COPY and RUN certutil steps to include your certificate names and paths. The version used in FROM should match the version of the synthetics-minion container being used. This version can be found in the docker logs output for a running synthetics-minion container or querying the minionBuildNumber attribute from the SyntheticsPrivateMinion event for your private location.

    FROM quay.io/newrelic/synthetics-minion-runner:3.0.62
    COPY ./myrootca.crt /opt/user/myrootca.crt
    COPY ./certificate.crt /opt/user/certificate.crt
    USER root
    RUN apt-get update
    RUN apt-get -y install libnss3-tools
    USER norbert
    RUN mkdir /opt/user/.pki/
    RUN mkdir /opt/user/.pki/nssdb
    RUN certutil -d sql:/opt/user/.pki/nssdb -N
    RUN certutil -d sql:/opt/user/.pki/nssdb -A -t "C,," -n myrootca -i /opt/user/myrootca.crt
    RUN certutil -d sql:/opt/user/.pki/nssdb -A -t "P,," -n myserver -i /opt/user/certificate.crt
    RUN certutil -d sql:/opt/user/.pki/nssdb -L
    

    Different -t options should be used with the certutil -A command based on the certificate type:

    • For root certificate authorities, use “C,”
    • For intermediate certificates, use “,”
    • For server / host certificates, use “P,”
  3. Build the custom Docker container from the Dockerfile.

    docker build --tag synthetics-minion-runner-with-certificate .
    
  4. Tag the new Docker image as synthetics-minion-runner with a version matching your synthetics-minion container and your Dockerfile (this command is a single line):

    docker tag synthetics-minion-runner-with-certificate:latest synthetics-minion-runner:3.0.48
    
  5. Start the synthetics-minion container using the docker run command provided on the Private Locations page and the version used in your Dockerfile and docker tag command, making sure to replace the private location key:

    docker run -e MINION_PRIVATE_LOCATION_KEY=abc123 -v /tmp:/tmp:rw -v /var/run/docker.sock:/var/run/docker.sock:rw -d --restart unless-stopped quay.io/newrelic/synthetics-minion:3.0.62
    

    This command may vary if custom node modules, Verified Script Execution (VSE), and other optional settings are configured.

  6. Execute your monitor and verify it is successful.

Additional Details - Downloading Certificates

Instructions will vary based on the browser used or if openssl is used. All potential options to retrieve the necessary certificates cannot be documented as part of this solution.

In Chrome on OSX, this can be done by:

  1. Navigating to the site
  2. Clicking the padlock or Not Secure indicator to the left of the address bar
  3. Click on Certificate in the drop down.
  4. Click on each level of the certificate then drag and drop the certificate to your desktop or another location

In Chrome on Windows, this can be done by:

  1. Navigating to the site
  2. Clicking the padlock or Not Secure indicator to the left of the address bar
  3. Click on Certificate
  4. Click on the Details tab, then Copy to File to export the certificate in DER or Base-64 encoded format
  5. Check the certification path tab to see if additional root / intermediate certificates need to be downloaded as well.

*Modifying the officially released CPM images means New Relic Support will not be able to assist in troubleshooting containers created from modified images should issues arise. As always, posts here in Explorers Hub are welcome, as are feature requests to make adding custom certificates easier without having to modifying Docker images.

1 Like