Feature Idea: Running private minions in AWS?

I want to set up a private location for Synthetics in my AWS account. Reading documentation, it looks like the only way to do that is to run a VM in a hypervisor. AWS EC2 doesn’t support that. Are there any alternatives to make this work on EC2?

https://docs.newrelic.com/docs/synthetics/new-relic-synthetics/private-locations/install-configure-private-minions

Thanks,
Andrey


New Relic edit

  • I want this, too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

_We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team

2 Likes

Hi there, if you have vmware in-house you can use their vCenter plugin to easily migrate the vm to EC2, details here: https://aws.amazon.com/ec2/vcenter-portal/

1 Like

Unfortunately I dont think that will really work for us… we have a security requirement that any application has to be deployed on top of our base AMI (which has encryption, certain security tools, etc). It would be great if this were available as an rpm.

@andrey_utis unfortunately our appliance requires a very specific software/hardware setup in order to run performant so it’s not as simple as an RPM install - at least not yet. With that said, does your security team make any concession for third party appliances?

Hello,
I was wondering if this is still the case at this time or has there been any developments in terms of making it more pluggable into AWS.

Thanks,
Ken

1 Like

I am wondering the same thing around compatibility with AWS. As the push to adopt a cloud first strategy moves forward (in my case with AWS), the need for this will grow.

1 Like

Hey @kenneth_leung & @brian.north

At the moment it still is the case that we provide and support only the OVF private minion file.

As @rferreira detailed above, it’s possible to migrate the minion VM to EC2, however any alterations to the base OVF file we provide are not supported by New Relic.

We can certainly get your interest in an AWS compatible private minion lodged as a feature request to the Product Developers. I’ll do that now, and will also add a poll to this forum topic - please do add your interest via that poll. :slight_smile:

interested this as well.

VOTE : YES - I am surprised NewRelic does not just have an OFFICIAL AMI in the AWS Marketplace. It would make more sense to publish an official AMI that users can use to spin up an EC2 Private Minion than to give them an RPM or Source or Binary to put on their own OS/EC2 because you can control all the requirements.

It should not be much harder to produce an official AMI as it is to produce and official OVF/VM.

An ami in the marketplace based on a simple linux distro would be ideal.

What is the current status on this? Has NR taken the great suggestions and added support for AWS?

Hey @kdavidson - Old fashioned private minions still come in the virtual machine image format (.ovf file) - however we do now offer CPMs, or Containerised Private Minions, which I know customers have had success getting into AWS - though I don’t believe we have AMIs with it pre-loaded.

Here’s the docs on that: https://docs.newrelic.com/docs/synthetics/new-relic-synthetics/private-locations/containerized-private-minion-cpm-configuration

1 Like

OK folks: We have gotten the CPM minions to run in AWS.
Below is the cloudformation script that will get a minion up an running. You’ll have to change a couple of the paramters (key id, vpc, subnets etc.) to get this template working in your environment.

The CPM is not a ‘well behaved’ docker file since it creates other docker containers, this negates compatibility with fargate or EKS, so you have to use ECS(and it’s EC2 requirement.) But that being said the CPM works well in the ECS world.

Keep in mind NR does not support this approach, we had to figure all of this out without their assistance, but here is what we came up with.

{
  "AWSTemplateFormatVersion":"2010-09-09",
  "Description":"AWS CloudFormation template, based on the sample at https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-ecs.html#quickref-ecs-example-1.json and modified to make work for our organization using NewRelic CPM.",
  "Parameters": {
      "NonExistentParameter": {
        "Type": "String",
        "Description":"This Cloudformation template is a basic setup, not too fancy.  Most of the changeable items are in the parameters section, mostly for documentation purposes in this example template.  This example is repetitive at times and not really what you'd likely put into production, it's meant to get you started only.",
        "Default": ""
      },
      "NonExistentParameter2": {
        "Type": "String",
        "Description":"AWS Command Line example: aws cloudformation create-stack --template-body  file://your-file.json --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --stack-name cool-stack-name",
        "Default": ""
      },
      "NonExistentParameter3": {
        "Type": "String",
        "Description":"Of course, you will probably run this in some sort of pipeline, but that is beyond the scope of what I will cover here.",
        "Default": ""
      },
      "AppName": {
        "Type": "String",
        "Description": "Specifies the name of the task.  This can be anyname you choose.",
        "Default": "NewRelic-CPM"
      },
      "ClusterName": {
        "Type": "String",
        "Description": "Specifies the name of the Cluster.  This can be anyname you choose.",
        "Default": "WebMonitor-Cluster"
      },
      "CPU": {
        "Type": "String",
        "Description": "This is used for both the required CPU and the CPU reservation.  ",
        "Default": "256"
      },
      "DesiredCapacity": {
          "Type":"Number",
          "Description":"Number of instances to launch in your ECS cluster.",
          "Default":"1"
      },
      "ImageURL": {
          "Type": "String",
          "Description": "This is the place where the Docker Image file is located, omit the http(s)",
          "Default": "quay.io/newrelic/synthetics-minion:latest"
      },
      "InstanceType": {
          "Type": "String",
          "Description": "Specifies the EC2 instance type for your container instances. Defaults to t2.large which seems to be about right for a couple of minions.\n",
          "Default": "t2.large",
          "ConstraintDescription": "must be a valid EC2 instance type."
      },
      "KeyName": {
          "Type": "String",
          "Description": "Optional - Specifies the name of an existing Amazon EC2 key pair to enable SSH access to the EC2 instances in your [non-prod?] cluster.\n",
          "Default": "your-keypair-if-desired-OR-empty"
      },
      "MaxSize":{
          "Type":"Number",
          "Description":"Maximum number of instances that can be launched in your ECS cluster.",
          "Default":"2"
      },
      "Memory": {
        "Type": "String",
        "Description": "This is used for both the required Memory and the Memory reservation.",
        "Default": "2048"
      },
      "MinionLogLevel": {
        "Type": "String",
        "Description": "This the value you wish for your Environment.",
        "Default": "DEBUG"
      },
      "MinionPrivateLocationKey": {
        "Type": "String",
        "Description": "This the value from YOUR NewRelic private minion.",
        "Default": "your-cpm-id"
      },
      "MyDesktopSSHIP":{
          "Type": "String",
          "Description": "Enter your IP in Cidr form:  1.2.3.4/32",
          "Default": "256.256.256.256/33"
      },
      "SubnetId": {
          "Type": "CommaDelimitedList",
          "Description": "Optional - Specifies the Comma separated list of existing VPC Subnet Ids where ECS instances will run\n",
          "Default": "your-subnet-1,your-subnet-2,your-subnet-3"
      },
      "VpcId": {
          "Type": "String",
          "Description": "Optional - Specifies the ID of an existing VPC in which to launch your container instances. If you specify a VPC ID, you must specify a list of existing subnets in that VPC. If you do not specify a VPC ID, a new VPC is created with atleast 1 subnet.\n",
          "Default": "your-vpc-1",
          "ConstraintDescription": "VPC Id must begin with 'vpc-' or leave blank to have a new VPC created\n"
      }
  },
  "Mappings":{
    "AWSRegionToAMI":{
        "nonexistentregion":{
            "comment1": "These ID's are as of 8/2019 and for the image amzn2-ami-ecs-hvm-2.0.20190709-x86_64-ebs",
            "comment2": "This images is an ecs optimized 2019 Amazon linux2."
        },
        "eu-north-1": {
            "AMIID": "ami-059aa04f0c253ad6b"
          },
          "us-east-1": {
            "AMIID": "ami-0fac5486e4cff37f4"
          },
          "us-east-2": {
            "AMIID": "ami-0dca97e7cde7be3d5"
          },
          "us-west-1": {
            "AMIID": "ami-0c6e63b58aac1048e"
          },
          "us-west-2": {
            "AMIID": "ami-0e5e051fd0b505db6"
          },
          "ap-south-1": {
            "AMIID": "ami-0a8bf4e187339e2c1"
          },
          "eu-west-3": {
            "AMIID": "ami-0d260f3e5ccd06043"
          },
          "eu-west-2": {
            "AMIID": "ami-0de1dc478496a9e9b"
          },
          "eu-west-1": {
            "AMIID": "ami-0ae254c8a2d3346a7"
          },
          "ap-northeast-2": {
            "AMIID": "ami-0accbb5aa909be7bf"
          },
          "ap-northeast-1": {
            "AMIID": "ami-04a735b489d2a0320"
          },
          "sa-east-1": {
            "AMIID": "ami-038707d64e5b8e7ba"
          },
          "ca-central-1": {
            "AMIID": "ami-01c07ee95e77abba8"
          },
          "ap-southeast-1": {
            "AMIID": "ami-05c6d22d98f97471c"
          },
          "ap-southeast-2": {
            "AMIID": "ami-039bb4c3a7946ce19"
          },
          "eu-central-1": {
            "AMIID": "ami-0650e7d86452db33b"
          }
    }
  },
  "Resources":{
    "AutoscalingRole":{
      "Type":"AWS::IAM::Role",
      "Properties":{
        "RoleName":{
          "Fn::Join":[
            "-",
            [
              {
                "Ref":"ClusterName"
              },
              "AutoscalingRole"
            ]
          ]
        },
        "AssumeRolePolicyDocument":{
          "Statement":[
            {
              "Effect":"Allow",
              "Principal":{
                "Service":[
                  "application-autoscaling.amazonaws.com"
                ]
              },
              "Action":[
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "Path":"/",
        "Policies":[
          {
            "PolicyName":{
              "Fn::Join":[
                "-",
                [
                  {
                    "Ref":"ClusterName"
                  },
                  "ServiceAutoscalingPolicy"
                ]
              ]
            },
            "PolicyDocument":{
              "Statement":[
                {
                  "Effect":"Allow",
                  "Action":[
                    "application-autoscaling:*",
                    "cloudwatch:DescribeAlarms",
                    "cloudwatch:PutMetricAlarm",
                    "ecs:DescribeServices",
                    "ecs:UpdateService"
                  ],
                  "Resource":"*"
                }
              ]
            }
          }
        ]
      }
    },
    "CloudwatchLogsGroup":{
      "Type":"AWS::Logs::LogGroup",
      "Properties":{
        "LogGroupName":{
          "Fn::Join":[
            "-",
            [
              {
                "Ref":"ClusterName"
              },
              "CloudWatchLogGroup"            ]
            ]
        },
        "RetentionInDays":14
      }
    },
    "ContainerInstances":{
      "Type":"AWS::AutoScaling::LaunchConfiguration",
      "Properties":{
        "LaunchConfigurationName":{
          "Fn::Join":[
            "-",
            [
              {
                "Ref":"ClusterName"
              },
                "LaunchConfiguration"
            ]
          ]
        },
        "ImageId":{
          "Fn::FindInMap":[
            "AWSRegionToAMI",
            {
              "Ref":"AWS::Region"
            },
            "AMIID"
          ]
        },
        "SecurityGroups":[
          {
            "Ref":"EcsSecurityGroup"
          }
        ],
        "InstanceType":{
          "Ref":"InstanceType"
        },
        "IamInstanceProfile":{
          "Ref":"EC2InstanceProfile"
        },
        "KeyName":{
          "Ref":"KeyName"
        },
        "UserData":{
          "Fn::Base64":{
            "Fn::Join":[
              "",
              [
                "#!/bin/bash -xe\n",
                "echo ECS_CLUSTER=",
                {
                  "Ref":"ECSCluster"
                },
                " >> /etc/ecs/ecs.config\n",
                "yum install -y aws-cfn-bootstrap\n",
                "/opt/aws/bin/cfn-signal -e $? ",
                "         --stack ",
                {
                  "Ref":"AWS::StackName"
                },
                "         --resource ECSAutoScalingGroup ",
                "         --region ",
                {
                  "Ref":"AWS::Region"
                },
                "\n"
              ]
            ]
          }
        }
      }
    },
    "EC2InstanceProfile":{
      "Type":"AWS::IAM::InstanceProfile",
      "Properties":{
        "InstanceProfileName":{
          "Fn::Join":[
            "-",
            [
              {
                "Ref":"ClusterName"
              },
                "InstanceProfile"
            ]
          ]
        },
        "Path":"/",
        "Roles":[
          {
            "Ref":"EC2Role"
          }
        ]
      }
    },
    "EC2Role":{
      "Type":"AWS::IAM::Role",
      "Properties":{
        "RoleName":{
          "Fn::Join":[
            "-",
            [
              {
                "Ref":"ClusterName"
              },
                "EC2Role"
            ]
          ]
        },
        "AssumeRolePolicyDocument":{
          "Statement":[
            {
              "Effect":"Allow",
              "Principal":{
                "Service":[
                  "ec2.amazonaws.com"
                ]
              },
              "Action":[
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "Path":"/",
        "Policies":[
          {
            "PolicyName":{
              "Fn::Join":[
                "-",
                [
                  {
                    "Ref":"ClusterName"
                  },
                    "ECSServicePolicy"
                ]
              ]
            },
            "PolicyDocument":{
              "Statement":[
                {
                  "Effect":"Allow",
                  "Action":[
                    "ecs:CreateCluster",
                    "ecs:DeregisterContainerInstance",
                    "ecs:DiscoverPollEndpoint",
                    "ecs:Poll",
                    "ecs:RegisterContainerInstance",
                    "ecs:StartTelemetrySession",
                    "ecs:Submit*",
                    "logs:CreateLogStream",
                    "logs:PutLogEvents"
                  ],
                  "Resource":"*"
                }
              ]
            }
          }
        ]
      }
    },
    "ECSAutoScalingGroup":{
      "Type":"AWS::AutoScaling::AutoScalingGroup",
      "Properties":{
        "AutoScalingGroupName": {
                  "Fn::Join":[
                    "-",
                    [
                      {
                        "Ref":"ClusterName"
                      },
                        "AutoScalingGroup"
                    ]
                  ]
        },
        "VPCZoneIdentifier":{
          "Ref":"SubnetId"
        },
        "LaunchConfigurationName":{
          "Ref":"ContainerInstances"
        },
        "MinSize":"1",
        "MaxSize":{
          "Ref":"MaxSize"
        },
        "DesiredCapacity":{
          "Ref":"DesiredCapacity"
        }
      },
      "CreationPolicy":{
        "ResourceSignal":{
          "Timeout":"PT15M"
        }
      },
      "UpdatePolicy":{
        "AutoScalingReplacingUpdate":{
          "WillReplace":"true"
        }
      }
    },
    "ECSCluster":{
      "Type":"AWS::ECS::Cluster",
      "Properties":{
        "ClusterName":{
          "Ref":"ClusterName"
        }
      }
    },
    "EcsSecurityGroup":{
      "Type":"AWS::EC2::SecurityGroup",
      "Properties":{
        "GroupDescription":"ECS Security Group",
        "GroupName":{
          "Fn::Join":[
            "-",
            [
              {
                "Ref":"ClusterName"
              },
              "SecurityGroup"
            ]
          ]
        },
        "VpcId":{
          "Ref":"VpcId"
        }
      }
    },
    "EcsSecurityGroupHTTPinbound":{
      "Type":"AWS::EC2::SecurityGroupIngress",
      "Properties":{
        "GroupId":{
          "Ref":"EcsSecurityGroup"
        },
        "IpProtocol":"tcp",
        "FromPort":"80",
        "ToPort":"80",
        "CidrIp":"0.0.0.0/0"
      }
    },
    "EcsSecurityGroupSSHinbound":{
      "Type":"AWS::EC2::SecurityGroupIngress",
      "Properties":{
        "GroupId":{
          "Ref":"EcsSecurityGroup"
        },
        "IpProtocol":"tcp",
        "FromPort":"22",
        "ToPort":"22",
        "CidrIp":{
            "Ref":"MyDesktopSSHIP"
          }
      }
    },
    "ECSServiceRole":{
      "Type":"AWS::IAM::Role",
      "Properties":{
        "AssumeRolePolicyDocument":{
          "Statement":[
            {
              "Effect":"Allow",
              "Principal":{
                "Service":[
                  "ecs.amazonaws.com"
                ]
              },
              "Action":[
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "Path":"/",
        "Policies":[
          {
            "PolicyName":{
              "Fn::Join":[
                "-",
                [
                  {
                    "Ref":"ClusterName"
                  },
                  "EcsServicePolicy"
                ]
              ]
            },
            "PolicyDocument":{
              "Statement":[
                {
                  "Effect":"Allow",
                  "Action":[
                    "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
                    "elasticloadbalancing:DeregisterTargets",
                    "elasticloadbalancing:Describe*",
                    "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                    "elasticloadbalancing:RegisterTargets",
                    "ec2:Describe*",
                    "ec2:AuthorizeSecurityGroupIngress"
                  ],
                  "Resource":"*"
                }
              ]
            }
          }
        ],
        "RoleName":{
          "Fn::Join":[
            "-",
            [
              {
                "Ref":"ClusterName"
              },
              "EcsServiceRole"
            ]
          ]
        }
      }
    },
    "Service":{
      "Type":"AWS::ECS::Service",
      "Properties":{
        "Cluster":{
          "Ref":"ECSCluster"
        },
        "ServiceName":{
          "Fn::Join":[
            "-",
            [
              {
                "Ref":"ClusterName"
              },
              {
                "Ref":"AppName"
              },
              "Service"
            ]
          ]
        },
        "DesiredCount":"1",
        "TaskDefinition":{
          "Ref":"TaskDefinition"
        }
      }
    },
    "ServiceScalingPolicy":{
      "Type":"AWS::ApplicationAutoScaling::ScalingPolicy",
      "Properties":{
        "PolicyName":"AStepPolicy",
        "PolicyType":"StepScaling",
        "ScalingTargetId":{
          "Ref":"ServiceScalingTarget"
        },
        "StepScalingPolicyConfiguration":{
          "AdjustmentType":"PercentChangeInCapacity",
          "Cooldown":60,
          "MetricAggregationType":"Average",
          "StepAdjustments":[
            {
              "MetricIntervalLowerBound":0,
              "ScalingAdjustment":200
            }
          ]
        }
      }
    },
    "ServiceScalingTarget":{
      "Type":"AWS::ApplicationAutoScaling::ScalableTarget",
      "DependsOn":"Service",
      "Properties":{
        "MaxCapacity":2,
        "MinCapacity":1,
        "ResourceId":{
          "Fn::Join":[
            "",
            [
              "service/",
              {
                "Ref":"ClusterName"
              },
              "/",
              {
                "Fn::GetAtt":[
                  "Service",
                  "Name"
                ]
              }
            ]
          ]
        },
        "RoleARN":{
          "Fn::GetAtt":[
            "AutoscalingRole",
            "Arn"
          ]
        },
        "ScalableDimension":"ecs:service:DesiredCount",
        "ServiceNamespace":"ecs"
      }
    },
    "TaskDefinition":{
      "Type":"AWS::ECS::TaskDefinition",
      "Properties":{
        "ContainerDefinitions": [
            {
                "Name": {"Ref": "AppName"},
                "Image": {"Ref": "ImageURL"},
                "Cpu": {"Ref": "CPU"},
                "MemoryReservation": {"Ref": "Memory"},
                "Essential": true,
                "Environment": [
                    {
                        "Name": "MINION_PRIVATE_LOCATION_KEY",
                        "Value": {"Ref": "MinionPrivateLocationKey"}
                    },
                    {
                        "Name": "MINION_LOG_LEVEL",
                        "Value": {"Ref": "MinionLogLevel"}
                    }
                ],
                "MountPoints": [
                    {
                        "SourceVolume": "tmp",
                        "ContainerPath": "/tmp"
                    },
                    {
                        "SourceVolume": "sock",
                        "ContainerPath": "/var/run/docker.sock"
                    }
                ],
                "VolumesFrom": [],
                "Privileged": false,
                "ReadonlyRootFilesystem": false,
                "LogConfiguration": {
                    "LogDriver": "json-file"
                },
                "HealthCheck": {
                    "Command": [
                        "curl -s -o /dev/null -I -w \"%{http_code}\" http://localhost:8080/status/check"
                    ],
                    "Interval": 180,
                    "Timeout": 30,
                    "Retries": 5,
                    "StartPeriod": 300
                }
            }
        ],
        "Cpu":  {"Ref": "CPU"},
        "ExecutionRoleArn": {"Fn::Sub": "arn:aws:iam::${AWS::AccountId}:role/ecsTaskExecutionRole" },
        "Family":{
          "Fn::Join":[
            "-",
            [
              {
                "Ref":"ClusterName"
              },
              "NewRelicCpmApp"
            ]
          ]
        },
        "Memory":  {"Ref": "Memory"},
        "NetworkMode": "bridge",
        "PlacementConstraints": [],
        "RequiresCompatibilities": [
            "EC2"
        ],
        "TaskRoleArn": {"Fn::Sub": "arn:aws:iam::${AWS::AccountId}:role/ecsTaskExecutionRole" },
        "Volumes": [
            {
                "Name": "tmp",
                "Host": {
                    "SourcePath": "/tmp"
                }
            },
            {
                "Name": "sock",
                "Host": {
                    "SourcePath": "/var/run/docker.sock"
                }
            }
        ]
      }
    }
  },
  "Outputs":{
    "ecsservice":{
      "Value":{
        "Ref":"Service"
      }
    },
    "ecscluster":{
      "Value":{
        "Ref":"ECSCluster"
      }
    },
    "taskdef":{
      "Value":{
        "Ref":"TaskDefinition"
      }
    }
  }
}
4 Likes

Hi Ryan:

is AWS ECS support in the works?. Thanks!

Hi @rajesh.rajasekharan - I’m not sure about the roadmap, I don’t have that visibility. But I can get your request for that submitted.

However for now I’d suggest trying out the information kindly shared by @kdavidson above, with the cloudformation script to get up and running in ECS.

Hi @kdavidson:

I’m facing an error during the ECS Task execution, when the “synthetic-minion” is trying to pull the “synthetic-minion-runner” image.

————

1591406414345 | 2020-06-06 01:20:14,345 [main] c.n.s.m.c.d.ContainerSystemDriverDocker INFO Docker Runner Image synthetics-minion-runner:3.0.1 does not exist: Trying to pull 9333236237862.dkr.ecr.us-west-2.amazonaws.com/synthetics-minion-repo/newrelic/synthetics-minion-runner:3.0.1… | 1591406414356 | 2020-06-06 01:20:14,356 [main] c.n.s.m.c.d.ContainerSystemDriverDocker WARN Docker Runner Image synthetics-minion-runner:3.0.1 was not able to be pulled from 9333236237862.dkr.ecr.us-west-2.amazonaws.com/synthetics-minion-repo/newrelic/synthetics-minion-runner:3.0.1. | 1591406414358 | 2020-06-06 01:20:14,357 [main] c.n.s.m.c.d.ContainerSystemDriverDocker DEBUG Full error message: ! com.spotify.docker.client.exceptions.DockerRequestException: Request error: POST unix://localhost:80/v1.35/images/create?fromImage= 9333236237862.dkr.ecr.us-west-2.amazonaws.com%2Fsynthetics-minion-repo%2Fnewrelic%2Fsynthetics-minion-runner&tag=3.0.1: 500, body: {“message”:“Get https://9333236237862.dkr.ecr.us-west-2.amazonaws.com/v2/synthetics-minion-repo/newrelic/synthetics-minion-runner/manifests/3.0.1: no basic auth credentials”}


As per the Cloudwatch log for the ECS Task, there is an Auth failure with the ECR when pulling the “synthetics-minion-runner” image. Can I know if you ran into a similar situation? .

In my scenario, the AWS environment is behind a proxy and cannot access resources from the Internet (like quay.io).

Thank you!

Hey Rajesh,

We did not have to use a proxy. But, there are other areas we don’t have access, for this we put a copy of outside resources in the AWS ECR (I don’t manage that part of our org, so I don’t have a lot of details there.). Sorry I couldn’t be of more help.

-kane

1 Like

Thanks Kane.

I was able to get it to work with the below “workaround” steps (not ideal).

Pull the Runner image to the ECS instance, and then tag it .

After the above steps were done in the ECS instance, then the ECS task execution was successful.

1 Like

That’s awesome! Glad you were able to make that work :smiley: