Could not fetch Muting Rule

We are getting a pretty serious error with our muting rules.
Most of them are now broken and we cannot edit them.
Clicking edit or trying to change their enabled state results in the error in the title “Could not fetch Muting Rule”
We don’t know if these are now muting live alerts. We assume they are.
We deployed today and our deployment process uses the New Relic API to enable and disable the muting rules in order to prevent alerts firing as the code is deployed.
All I can think is that this has somehow left our muting rules in an invalid state.
Can someone look at this please with some urgency? It has potentially broken our live system and has left us vulnerable and unable to correctly monitor our production systems.

The muting rules for our account seem really screwed up. It looks like they’ve somehow duplicated and most of them are currently unusable. I cannot view, edit or delete them.
How might I raise a support ticket to get these fixed or deleted?
I’d like to pass along our account so someone can take a look

Can I get a response to this please?
Or get a support ticket started?

Can we pay for support?

Feels pretty bad to not be able to even talk to someone about something that seems truly broken in New Relic and is potentially impacting the visibility of our live systems.

I’ve also attempted to query and delete these muting rules via the GraphQL API.
When listing them it shows 9 of the rules are duplicated with the same id and name. That cannot be a good thing.
When attempting to delete any of these muting rules again using the GraphQL command we receive an Internal Server Error.

Any chance we can get this triaged as a bug? Seems pretty broken to me.
I posted this issue on Thursday (evening UK time granted) but I’ve had no reply to this so far… I’m becoming quite frustrated at the lack of response. Can I at least get some acknowledgement of my problem from NR support?

@NRcommunityteam Sorry to tag but can I get some help please?

@nmcnamara @kmullaney @cfrankenfield
I’m really sorry to tag you here but I don’t know what else to do to get any response to this.
You all recently helped me on other posts so I am hoping one of you can jump in here.
I’m at my wits end on this one and just need someone to acknowledge.
The silence has been deafening and I am frustrated to the point I don’t even know if I am overreacting any more! :sob:

@mike.smith Sorry to hear about your frustration. Our support team has been notified of this issue and will respond as soon as they are able.

Hi @mike.smith - Sorry you have been waiting so long for a response. Sounds like you are unable to use muting rules at all. Can you try using an incognito browser and disabling any browser add ons? If you have already tried that please provide a link to the muting rules on your account and I’ll be happy to take a look.

1 Like

Thank @dkoyano

I just tried an incognito window in a new browser but that had the same issue. Like I said above it seems to be broken at a server level as querying the NR GraphQL API shows the duplicated IDs as well as server errors when trying to delete.

I can’t provide you a specific permalink to the broken rules because I can’t get into them to edit and click the permalink button.

The permalink for my list of muting rules is https://one.newrelic.com/-/0znQxE1zXwV

The manually compiled list I have of rules I know to be broken is as follows:

ID      Name
17494	Health Alerts for B2B UAT
17495	Performance Alerts for B2B UAT
17496	Health Alerts for B2C UAT
17498	Health Alerts for API UAT
17499	Performance Alerts for API UAT
17510	Health Alerts for B2C Live
17511	Performance Alerts for B2C Live
17512	Health Alerts for API Live
17513	Performance Alerts for API Live

Ideally if we could deduplicate these and get them working again that would be ideal, but if that is not possible and they need to be deleted then that is also fine as I should be able to recreate them.

If you need a reproduction of what (I think) I did to break them in the first place let me know and we can look at that after these are repaired.

For your reference the GraphQL query I used is:

{
  actor {
    account(id: <account>) {
      alerts {
        mutingRules {
          id
          name
          enabled
        }
      }
    }
  }
}

A snippet of the result clearly shows the duplicated rules:

...
            {
              "enabled": true,
              "id": "17498",
              "name": "Health Alerts for API UAT"
            },
            {
              "enabled": true,
              "id": "17498",
              "name": "Health Alerts for API UAT"
            },
            {
              "enabled": true,
              "id": "17498",
              "name": "Health Alerts for API UAT"
            },
...

To clarify, there are some muting rules in the list that I can continue to use. These are either created originally and just didn’t duplicate and break, or I have created some later and they are still functional.

Please let me know if you require any more information.

@dkoyano Hello!
Any update on this?

Hi,

Any update on this? Is it being looked into?
We still cannot effectively use our muting rules and live alerts.

@dkoyano
Any update on when my response might be looked into further?
Can we open a ticket with a technical or development team?
Has it already been moved forward and I’ve just not been informed?

Any update at all would be appreciated.

@dkoyano Are you serious?

I immediately provided the information you requested and I’ve received no response.

We’re approaching the 3 week mark here since I first reported this problem and I have heard nothing that suggests it is being fixed or investigated.
Our muting rules are still completely broken and potentially active and suppressing live alerts.

@JoiConverse Can I please get some more support here. Even just a response. Anything.

STILL
WAITING
FOR
A
RESPONSE

Is it being looked at? Am I wrong? What’s going on?

@nmcnamara @kmullaney @cfrankenfield @JoiConverse @dkoyano @NRcommunityteam

Sorry not sorry for the tags now because that was seemingly the only way I actually got an answer from you 3 WEEKS AGO!

I know you are supposedly short on staff this week with some event, but seriously… 28 days since the original report of a bug with the platform and there’s NO meaningful response?

@mike.smith I understand your frustration and our support team are following up on responses as they can. We will followup with them again. You have been patient thus far, thank you!

Hi @mike.smith - I sincerely apologize for the delay. Can you provide an account number? I wasn’t able to see any muting rules on the account when going to the link you provided. I will be watching for your reply and will respond once I get the account number and am able to take a look at a few things.

I’ve been on holiday hence my delay.
Account number is 3018686
It’s an EU account.

https://onenr.io/0rVRVr4gMRa
Muting rule list is here again although it’s confusing why the previous short link didn’t work…

Anything else you need please let me know and I will try and check back in when I can.

Hi @mike.smith - I was able to reproduce this so I am going to create a support ticket in order for an Engineer to take a look and find out what’s going on here.