Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

New Violation Time Limit setting?


#1

I was just doing a review of my Infrastructure alert conditions and noticed a new “Violation Time Limit” setting at the bottom of the page which didn’t used to be there. I’m a bit concerned because it looks like all of my alerts have been given a “Close open violations after 24 hours” setting now, even though I never set this setting myself. I don’t want any of my alerts closed automatically.

Can someone tell me when this setting was added, why it was set to something other than “never” (which seemingly should be the default), and how to now set all of my alerts to “never”?

Alek


#2

Hi @Hayorti, this is a new feature for conditions like Host Not Reporting, and although when you edit the condition the it looks like after 24 hours is set as default, the default behaviour in the backend is Never. So unless you change the option in that dropdown the settings of your condition won’t change :slight_smile:


#3

That’s concerning, to be honest. How am I supposed to know which ones have been knowingly set to after 24 Hours and which ones are actually Never? On the same note, if I change the conditions of an alert and save them, how am I supposed to know if the system’s also changed this value from a “fake” after 24 hours to a real one?


#4

You’re right it is confusing, i’ve raised this with our Product Development team who are looking at changing the wording on existing conditions, as well as changing the default Dropdown option to whatever the condition is set to.


#5

So, I just received the email below from service_notifications@newrelic.com. I remember reading the “original announcement”, but I revisited it anyway, and confirmed that it never actually states that existing conditions will be changed. It simply says, “Accounts migrated to the new pipeline will default to a 24hr force-close for violations.” Also, how are we supposed to now go through 246 conditions and change this value? I’m definitely not doing it through the UI. So, am I expected to make 246 API calls now to bulk-reverse this change? I still don’t understand why this bulk change was made without people’s knowledge.

I would still like to have this setting reversed without having to make ~250 API calls.

Hello!
We are reaching out today to let you know about a new configuration option in New Relic Infrastructure Alerting. As we standardize behavior across our alerting platforms, and as mentioned in the original announcement, we are introducing a new condition configuration to allow you to control how long to wait before force-closing open violations.

What’s changing?

  • All existing Infrastructure alert conditions will default to close 24 hours after they are opened.
  • If the violation is the last one in the associated incident, the incident will close as well.
  • If the alert condition is still matched; the violation will reopen and a new incident will be created.

How do I change it from the default?
You can enable/disable the feature and the duration through the New Relic Infrastructure UI or API.

UI

  • A new section “Violation time limit” has been added to the alert condition edition
  • All existing and new alert conditions will default to force-close 24 hours after they are opened
  • The Violation time limit can be deactivated so that violations will stay open until
    • The alert condition is no longer met
    • The violation is manually closed

API

  • A new attribute is available when creating or editing an alert condition via the API:
    • violation_close_timer: defines the time period before an alert violation is closed (in hours)
  • if violation_close_timer is set to 0 , the timer for the alert condition will be disabled
  • if violation_close_timer is set to > 0 , the timer will be set to this value
  • If this attribute is not present in the request; a default value will be configured
    • violation_close_timer: 24

What you may notice:
As alerts are migrated to the new pipeline, a default violation time of 24 hours will be set. Any violations that have been open for longer than 24 hours will be force-closed. A new violation will be opened if the condition is still in violation and a new notification will be sent.

Please see the Explorer’s Hub post for additional information.

Best regards,
Your New Relic Infrastructure and Alerting Teams


#6

Hi, @Hayorti: First, let me say that I understand your apparent frustration at having your existing alert conditions changed without your permission; this reply is not meant to minimize it. I am not on the team that made that decision, so I have no ability to change or explain it.

I am, however, a member of the Customer Solutions Group, whose job is to help customers use New Relic products to their full potential. In that vein, here is a solution (for you, as well as others who may discover this thread while looking for a solution): I have written a Python script that loops through your Infrastructure alert conditions and sets their violation close timer to 0 (never). Just replace the string ‘your admin key’ with your New Relic admin API key:

import requests
import json

ADMIN_API_KEY = 'your admin key'

headers = {
  'X-Api-Key': ADMIN_API_KEY, 
  'Content-Type': 'application/json'
}

# Set violation close timer to 0 (never)
update = {
  'data': { 
    'violation_close_timer': 0
  }
}

limit = 50
offset = 0

while True:
  url = 'https://infra-api.newrelic.com/v2/alerts/conditions?limit={}&offset={}'.format(limit, offset)

  # Get Infrastructure alert conditions
  response = requests.get(url, headers = headers)
  conditions = response.json()
  conditions_returned = len(conditions['data'])
  
  # Loop through conditions, updating violation close timer
  for condition in conditions['data']:
    url = 'https://infra-api.newrelic.com/v2/alerts/conditions/{}'.format(condition['id'])
    response = requests.put(url, data = json.dumps(update), headers = headers)
    print(response.status_code)
    
  # If no more conditions, exit loop
  if conditions_returned < limit:
    break
  else:
    # Otherwise, get next batch
    offset += limit