Extract with Synthetic monitor data from New Relic API in Python

Hello,
I am unable to retrieve data from the New Relic API (‘https://synthetics.newrelic.com/synthetics/api/v3/monitors’). When I execute the GET method request, the API returns me only 46 monitors out of more than 1500.

Hi, @pat10: Please post the code you are using to get your monitors.

url = 'https://synthetics.newrelic.com/synthetics/api/v3/monitors'
api_key = 'NRAK-492xxxxxxxxxxE'

headers = {
    # 'Accept': 'application/json',
    'X-Api-Key': api_key,
    'Content-Type': 'application/json'
}

def extract_nr_data(url, api_key):
    response = requests.get(url, headers=headers)
    data = json.loads(response.text)
    monitors = data["monitors"]
    count=0
    list_url_newrelic = []
    for elt in monitors:
        if elt['type'] == monitorType:
            list_url_newrelic.append(elt['name'])
            count=count+1
    print("Count url to extract from NR: ",count)
    return list_url_newrelic

I don’t see where monitorType is defined? In any case, REST API results are paginated. You may pass a limit parameter to return up to 100 results per call. If you have more than 100 monitors, you must check the response header to see if there are additional pages of results:

response = requests.get(url, headers = headers)
while 'next' in response.links.keys():
  response = requests.get(response.links['next']['url'], headers = headers)

You might also consider using the NerdGraph API to execute an NRQL query, which can return up to 2000 results per query.

Hi Phil,
I send you all the code in Python to extract via the NerdGraph API all the monitors created. This code just returns me 200 records out of 1600. I read in the New Relic docs (https://docs.newrelic.com/docs/apis/nerdgraph/examples/nerdgraph-entities-api-tutorial/) which doesn’t did not help me how to do with nextcursor. Thanks for the help.

from asyncio.log import logger
import requests
import json
from license import user_key
import logging

def nerdgraph_readtags(key):
  # GraphQL query to NerdGraph
  query = """  
  {
    actor {
      entitySearch(query: "domain = 'SYNTH' AND type = 'MONITOR'", sortBy: NAME) {
        results {
          entities {
            name
          
          }
        }
        count
      }
    }
  } """


  # NerdGraph endpoint 
  endpoint = "https://api.newrelic.com/graphql"
  headers = {'API-Key': f'{key}'}
  response = requests.post(endpoint, headers=headers, json={"query": query})
  
  while 'next' in response.links.keys():
      response = requests.get(response.links["data"]["actor"]["entitySearch"]["results"]["entities"], headers = headers)
  #print(response.content)  

  
  if response.status_code == 200: 
    # convert a JSON into an equivalent python dictionary
    dict_response = json.loads(response.content)
    #print(dict_response["data"]["actor"]["entitySearch"]["results"]["entities"])

    monitors=dict_response["data"]["actor"]["entitySearch"]["results"]["entities"]
    list_url_newrelic = []
    count=0
    for elt in monitors:
      list_url_newrelic.append(elt['name'])
      count=count+1
    print("Count url to extract from NR: ",count)
    
    print(list_url_newrelic)
    
    # optional - serialize object as a JSON formatted stream 
    json_response = json.dumps(response.json(), indent=2)
    #print(json_response)
  else:
    # raise an exepction with a HTTP response code, if there is an error
    raise Exception(f'Nerdgraph query failed with a {response.status_code}.')

nerdgraph_readtags(user_key)

Hi @pat10

Humm let me get a support engineers eyes on this, as this is above my scope. Please note they will reach out via this post with their findings.

@pat10 - You should be able to select nextCursor under the results section as shown in the screenshot below.

1 Like

Hi, thanks for your answer, but my problem lies in extracting all monitors from SSL certificates. I would like to know if there is a way recommended by New Relic to extract the list of tests from an account. It might help. I thought I could do it with Python, but only, only 200 records are fetched out of more than 1600. As, you can see the presented script gives the result below:
This is a snippet of the output from the script that counts the number of tests fetched:

Count url to extract from NR: 200
['0009xxxxxd35018.hp.dakis.io', '00b1xxxxxbb30eb.hp.dakis.io', '00d1xxxxx6c0000.hp.dakis.io', '0164xxxxx69e032.de.dakis.io', '01caxxxxx1640c0.hp.dakis.io' , '023cxxxxx2f600c.hp.dakis.io', '02ecxxxxx4c017.hp.dakis.io', '03aaxxxxx2f90f2.de.dakis.io', '03bdxxxxx7ad0f3.de.dakis.io', ....

Without however asking for a correction of my script, I seek help to find the missing instructions or how to modify the script. Furthermore, I would be interested in an example of code recommended by New Relic, whatever the language, likely to extract all the tests.

Also, unless I am mistaken, the GraphQL query you sent me contains an error.

Thanks for your help

As I replied above:

You might also consider using the NerdGraph API to execute an NRQL query , which can return up to 2000 results per query

{
   actor {
      account(id: YOUR_ACCOUNT_ID) {
         nrql(query: "SELECT uniques(monitorName) FROM SyntheticCheck SINCE 1 day ago LIMIT MAX") {
            results
         }
      }
   }
}

The above query will return the names of monitors that have run within the past 24 hours. If you also need the names of inactive monitors, you will have to do an entity search and use nextCursor to retrieve multiple pages.

Hi, thank you for your answer, it helped me a lot and see the power of NRQL. Only, I managed to extract a maximum of 1000 records out of the approximately 1600.
The goal is to write a script capable of periodically updating the list of active monitors via the New Relic API. A comparison of the lists of monitors coming from our and the new relic api will identify the monitors to be deleted and the new monitors to be created. It is therefore essential that our script can extract all the monitors in use at new relic.
I share with you the results of my tests:
The execution of the query below is limited to 1000 records extracted.

{
  actor {
    account(id: xxxxxx) {
      nrql(query: "SELECT unique(monitorName) FROM SyntheticCheck SINCE 1 day ago LIMIT MAX") {
        results
      }
    }
  }
}

Changing the parameters SINCE (SINCE 5 week ago) and **LIMIT ** (LIMIT 2000 or LIMIT 600) I always get 1000 extracted records out of 1648.
The following query counts the total number of monitors of 1648.

{
  actor {
    entitySearch(query: "domain = 'SYNTH' AND type = 'MONITOR'", sortBy: NAME) {
      count
    }
  }
}

I’m having trouble understanding why not all records are extracted.

Thanks again for your help

Hey @pat10
When you run a uniques() query, the max results you can obtain is 10k. And you set that limit within the brackets of the uniques query. Something like this should work for you:

{
  actor {
    account(id: xxxxxx) {
      nrql(query: "SELECT uniques(monitorName, 2000) FROM SyntheticCheck SINCE 1 day ago") {
        results
      }
    }
  }
}

I just tried this on my own account in my own data set, and I managed to retrieve 2935 rows of data via GraphQL.

Let us know if that doesn’t work for you.

1 Like

Hi, the nrql query helped me a lot. I have two more requests for you please.

  • I would like to refine my selection to get the name of the monitor , Entity guid and Monitor ID by modifying the request that you proposed to me or better did you request able to delete a monitor with NerdGraph.
  • Also, how could I get the list of fields from SyntheticCheck.

Really thanks for the help.

How could I get the list of fields from SyntheticCheck

https://docs.newrelic.com/attribute-dictionary/?event=SyntheticCheck

Hello,
Thanks for the help.
With the script I got to remove a large number of monitors (800). But, I realized that after deleting these monitors. The query below does not give an accurate count of active monitors on new relic: In the query result list, I can still see deleted monitors.

{
  actor {
    account(id: xxxxxxx) {
      nrql(query: "SELECT unique(monitorName, 2000) FROM SyntheticCheck SINCE 1 day ago") {
        results
      }
    }
  }
}

On the other hand, with the query below, I get an exact list but it only returns 200 records.

{
  actor {
    entitySearch(query: "domain = 'SYNTH' AND type = 'MONITOR'") {
      count
      results {
        entities {
          name
        }
      }
    }
  }
}

For my Python script, I need the query to return all records with NerdGraph via the https://api.newrelic.com/graphiql endpoint.

How to do that with this query?

Thanks again for the help. With this, I progress in my learning of New Relic.

The first query returns the list of monitors that have run in the past 24 hours. If you have deleted them within the past day, they will still appear in the result.

To use the second query, you must use nextCursor to request multiple pages.

Hi, I’m having trouble using nextcursor to fetch all the data in one query.

First request: It generates a value in nextcursor to be used in the second request.

Second query: It returns nothing as data.

How can this process be used in a Python script? Do you have a sample Python code (or query) that shows how to use nextcursor with NRQL? It will help me. The New Relic documentation does not show enough how to use nextcursor.

Thanks again for your answers.

There is an example on the page I have linked to (twice):

I cannot tell you any more than that.

Your second query returns no results because it is searching for "name like 'nerd-graph'". You should execute the same query as your first one: "domain = 'SYNTH' and type = 'MONITOR'".

Oh sorry. An error has occurred in the request. It works. But, I’m still trying to find an example code in Python to extract all the data …