NewRelic Synthetics PING MULTIPLE URLs

I want to ping multiple URLs. Here is an idea what I want to do:
My php code… The $Url variable will get several urls and the result must get the http code status that’s all. I want a perfect report in new relic synthetics. How can i achieve this???

$url = 'http://magento.local/index.php/default-subcategory-151125681.html';
$Response=substr(get_headers($url)[0], 9, 3);
if ($Response !== null) {
    switch ($Response) {
        case 100: $text = 'Continue'; break;
        case 101: $text = 'Switching Protocols'; break;
        case 200: $text = 'OK'; break;
        case 201: $text = 'Created'; break;
        case 202: $text = 'Accepted'; break;
        case 203: $text = 'Non-Authoritative Information'; break;
        case 204: $text = 'No Content'; break;
        case 205: $text = 'Reset Content'; break;
        case 206: $text = 'Partial Content'; break;
        case 300: $text = 'Multiple Choices'; break;
        case 301: $text = 'Moved Permanently'; break;
        case 302: $text = 'Moved Temporarily'; break;
        case 303: $text = 'See Other'; break;
        case 304: $text = 'Not Modified'; break;
        case 305: $text = 'Use Proxy'; break;
        case 400: $text = 'Bad Request'; break;
        case 401: $text = 'Unauthorized'; break;
        case 402: $text = 'Payment Required'; break;
        case 403: $text = 'Forbidden'; break;
        case 404: $text = 'Not Found'; break;
        case 405: $text = 'Method Not Allowed'; break;
        case 406: $text = 'Not Acceptable'; break;
        case 407: $text = 'Proxy Authentication Required'; break;
        case 408: $text = 'Request Time-out'; break;
        case 409: $text = 'Conflict'; break;
        case 410: $text = 'Gone'; break;
        case 411: $text = 'Length Required'; break;
        case 412: $text = 'Precondition Failed'; break;
        case 413: $text = 'Request Entity Too Large'; break;
        case 414: $text = 'Request-URI Too Large'; break;
        case 415: $text = 'Unsupported Media Type'; break;
        case 500: $text = 'Internal Server Error'; break;
        case 501: $text = 'Not Implemented'; break;
        case 502: $text = 'Bad Gateway'; break;
        case 503: $text = 'Service Unavailable'; break;
        case 504: $text = 'Gateway Time-out'; break;
        case 505: $text = 'HTTP Version not supported'; break;
        default:
            exit('Unknown http status code "' . htmlentities($code) . '"');
            break;
    }
}
    print_r('This URL ' . $url.' returns a response '. $text);
    if (intval($Response)>305)
    {
        //write to file-- bad links
        $myfile = fopen("BadLink.txt", "w") or die("Unable to open file!");
        $txt = ' ' . $url.' http status '. $text.'  '. $Response;
        fwrite($myfile, $txt);
        $txt = "\n";
        fwrite($myfile, $txt);
        fclose($myfile);
    }
    else 
    { //write to file-- Good link
        //write to file-- bad links
        $myfile = fopen("GoodLink.txt", "w") or die("Unable to open file!");
        $txt = ' ' . $url.' http status '. $text.'  '. $Response;
        fwrite($myfile, $txt);
        $txt = "\n";
        fwrite($myfile, $txt);
        fclose($myfile);

Hi @jahved_cassim

Can I confirm my understanding of your question? If I read correctly, you have a php page which will provide the statuses as listed in the switch/case and the test should pass if you receive a response matching the switch/case?

Or are you looking to load the page and check the information on it?

1 Like

Hi Stefan,

The second part … just ping the URLs and get the http status code

What I need is we provide new relic synthetics with several urls for example all the URLs found a web page sitemap and we get a report which displays the http status code for each of these URLs.

The code which i have written is taking the Urls and returning the http status code… get_headers($url)

Is there any script which does that in New Relic Synthetic. We want to test several urls at once.

Regards
Jahved

In that case it seems that you require a browser test to get the url’s from the page and then make the required request.get and put the logic in to fail or pass the test based on the results given.

Yes, what we can also do is to read the URLs from an external file(text file or database) and give us the http status code of each Urls in a fantastic report.

Is that possible?

Not really what synthetics are for but should be possible. I think that if you want to write a file then you would need to have another host to put it on.

ok … let’s forget about the file. Is it possible to get the HTTPS code status for let’s say 3 urls at once.
google.com get http code 200
me.com give http code 500
test.com give http code 400

Yes, you will need to create an API synthetic with a script something along the lines of:

var assert = require('assert');
var options = {}; // put your options including the url here

$http.get(options, function(error, response, body){
    if (error)
    {
      console.log(error); // fail the test here if required
    }
    else
    {
      // Verify endpoint returns 200 (OK) response code.
      // will fail test if response code is not 200
      assert.ok(response.statusCode == 200, 'Expected 200 OK response, actual:' + response.statusCode);
    }
  });

Repeat for each url you want to test.

2 Likes

All this is possible and we’re doing exactly this in some cases to workaround the limitation of not being able to create synthetics via an API.

There’s a few things to be mindful of when going down this path:

  • Each synthetic must complete in less than 3 minutes - you won’t be able to use this to monitor 1000’s of endpoints in one go but a few handfuls will be fine.
  • If a single web site fails or times out, having this “super monitor” fail natively may not give you suitable granularity in the alert. You probably want to know “X failed”, “Y exceed the timeout” as separate alerts. One approach to doing this is make sure the “super monitor” doesn’t fail, push individual results into a custom synthetics table and then scrape that data out via API’s into your chosen monitoring product.
  • Given you’re posting custom data into insights, for these alerts, you may well want to pass in more custom app/env/org-centric metadata in here to give better context to the alerts. This also allows setting up of alert routing or criticality etc within your end product.
  • Don’t blindly trust the https://insights-collector.newrelic.com/v1/accounts/ API - it will accept invalid JSON and throw it away while giving you a 200 response!

Below is a snippit I’ve chopped down from a SOAP service where we want to monitor the load balancer and the underlying nodes. In this case the JSON spec of what to monitor is inline but you could easily pull this back via an API call or a simple request of a web server.

var assert = require('assert');
var Joi = require('joi');

// For a new subaccount, update the account ID in the URL and generate a new insights key under Insights > Manage Data > API Keys
var myAccountID = '111111';
var myInsertKey = 'RepalceThisWithYourAPIKEY';

// The following schema defines a valid configuration array.  Adjust this to whatever data you want to post into insights.
var configschema = Joi.array().items(Joi.object().keys({
    endpoint: Joi.string().uri().required(),
    product: Joi.string().required(),
    environment: Joi.string().regex(/^(prod|dr|ebf|uat|qa|cat|perf|test|sit|dit|source|dev)$/).required(),
    type: Joi.string().required(),
    shortname: Joi.string().required(),
    criticality: Joi.string().regex(/^(CRITICAL|WARN|ERROR)$/).required()
}));

// This defines what the new relic alerts API will accept as post data i.e multiple events.
//  We only ever post a single event at a time.
var eventschema = Joi.array().items(Joi.object().keys({
    eventType: Joi.string().required(),
    guid: Joi.string().required(),
    location: Joi.string().required(),
    endpoint: Joi.string().required(),
    product: Joi.string().required(),
    environment: Joi.string().required(),
    shortname: Joi.string().required(),
    testtype: Joi.string().required(),
    criticality: Joi.string().required(),
    responsetime: Joi.number().required(),
    teststatus: Joi.string().required()
}));

var DefaultTimeout = 1000;

var monitorconfig = [{
    "endpoint": "https://f.q.dn/path/service",
    "product": "Product1",
    "environment": "prod",
    "type": "load balancer",
    "shortname": "Prod Load balanced widget"
    "criticality": "CRITICAL"
}, {
    "endpoint": "https://f.q.d.n.2/path/service",
    "product": "Product1",
    "environment": "prod",
    "shortname": "Prod widget node 2"
    "type": "node",
    "criticality": "WARN"
}, {
    "endpoint": "https://f.q.d.n.3/path/service",
    "product": "Product1",
    "environment": "prod",
    "shortname": "Prod widget node 3"
    "type": "node",
    "criticality": "WARN"

}];
var SoapRequestXML = '< ... soapy goodness goes here </soap:Envelope>';

function createcustomlog(insightdata, mytime, mystatus) {
     var insightpostoptions = {
        // We're running this from private minions so will need to make sure insights POSTs go out via the proxy
	proxy: "http://my.internal.proxy:1234",
        url: "https://insights-collector.newrelic.com/v1/accounts/" + myAccountID + "/events",
        // All these events will be posted into the  "dummy" custom events table:
        body: '[{"eventType": "dummy","guid": "' + $env.JOB_ID + '","location": "' + $env.LOCATION + '","endpoint": "' + insightdata.endpoint + '","product": "' + insightdata.product + '","environment": "' + insightdata.environment + '","shortname": "' + insightdata.shortname + '","testtype": "' + insightdata.type + '","criticality": "' + insightdata.criticality + '","responsetime": ' + mytime + ',"teststatus": "' + mystatus + '"}]',
        headers: {
            'X-Insert-Key': myInsertKey,
            'Content-Type': 'application/json'
        }
    };
    console.log("Validating insights API data to post an event for " + insightdata.endpoint);
    Joi.validate(insightpostoptions.body, eventschema, function(err, value) {
        assert(!err, "Couldn't validate constructed JSON record to post into insights" + err);
    });
    console.log("data to upload into insights looking good...");
    $http.post(insightpostoptions, function(error, response, body) {
        if (error) {
            console.log("Cannot post data to insights.  Got error: " + error);
        } else if (response.statusCode == 200) {
            console.log("OK: results posted to insights.  Response was: " + body);
        } else {
            console.log("OK: results not posted to insights.  response:" + response.statusCode + ", body: " + body);
        }
    });
  console.log("...");
}

function servicemonitor(thisendpoint) {
    var testStatus = 'UNKNOWN';
    var options = {
        url: thisendpoint.endpoint,
        body: SoapRequestXML,
        timeout: DefaultTimeout,
        headers: {
            'Content-Type': 'application/soap+xml;charset=UTF-8;action="urn:PokeTheWidget"'
        }
    };
    var testStartTime = new Date();
    $http.post(options, function(error, response, body) {
        var responsetime = (new Date() - testStartTime).toFixed(0);
        if (!error && response.statusCode == 200) {
            testStatus = 'SUCCESS';
        } else {
            testStatus = 'FAILED';
        }
        console.log(thisendpoint.shortname + " returned " + testStatus + " in " + responsetime + " ms");
        createcustomlog(thisendpoint, responsetime, testStatus);
    });
}

console.log("Validating monitoring configuration");
Joi.validate(monitorconfig, configschema, function(err, value) {
    assert(!err, "Couldn't validate configuration array in monitor" + err);
});
console.log("Monitoring configuration OK");
monitorconfig.forEach(function(configobj) {
    servicemonitor(configobj);
});
3 Likes

Thanks for that @pault most helpful.

WOW!!! Everyone here - I am blown away by the interaction and help here. Thank you @stefan_garnham and @pault for bringing your incredible expertise to this thread and sharing your time to help out.

1 Like

Thanks Pault & Stefan, this really help.

1 Like