Sounds great. Thanks!
Hey, @hross – just add some fuel to the fire.
I was preparing to go live with New Relic in our environment by June 30th and my entire alerting schema was dependent on NRQL-based alerts that used FACET in the query generating unique Incidents (and triggering the associated Notification Channels). After some testing today, I realized that FACET is not respected even when the policy type is set to By Condition and Entity.
I assume that is expected given the discussion above?
I’ve included a sample NRQL query I am using in my alerts below:
SELECT UNIQUECOUNT(processDisplayName) FROM ProcessSample FACET entityName WHERE ((entityName like '%OKTAG%' AND entityName NOT LIKE 'WP%')) AND processDisplayName = 'OktaAgentService.exe'
We did this because, as @zackm points out, if we try and create conditions that do complex filtering via the Infrastructure API, it blocks us from using the UI to do updates. Since our team sets the standards and app teams manage them for their specific team, we didn’t want to make those app teams dependent on learning and using the API to be successful.
Totally understood, and sorry that this has been so complicated. It’s a new feature, and we’re still working out what’s going on here. Turns out, ya’ll are so eager for it we’re pushing the boundaries from day 1. I believe we’re still working a ticket with us, and I will definitely make sure we share the outcome here.
Any ETAs? I’ve sent a note to our account manager and Alec Isaacson for visibility. Unfortunately, this is blocking my go-live.
Any information will help!
Yikes. I am literally watching our support and product teams talk this out in a chat session, so hopefully soon, but no guarantees at the moment. Suffice tit to say that we are all so invested in figuring this out on the support side that we are working this angle as hard as we can.
The June New Relic Newsletter made me sad I was so excited to see the below statement thinking it was fixed…and then realized it was the marketing email gods mocking me!
FACET Keyword Now Accepted on NRQL Alert Conditions
Did you know that you can set alert conditions based on NRQL queries in New Relic Insights? FACET has been added to the supported syntax for these queries—here are details on the FACET clause, and here’s how you can create alert conditions.
Pro Tip: You can always visit our docs site for the latest release notes and agent updates.
I am so sorry this has been an emotional roller coaster! Current status - Engineering and support are pairing today to try to wrap our arms around it. It’s not just you folks who have reported issues, and they want to look at everything together. Here’s what I know now:
- It is our intention for faceted NRQL alerts to work in the way that you are trying to make them work
- It is unclear why they are not
- It is highly unlikely that it will be resolved in time for your June 30 go live. Anything is POSSIBLE, but even if they find the problem today, we can’t guarantee how quickly we can resolve the issue.
I know that’s not good news, but I want to be as up-front with you as I can.
There is a workaround that may be a viable Plan B for you, which is to create an alert condition PER facet. Yes - this is a lot more work. But if you’re only needing to alert on a few facets, it’s an option. If your facets have 100+ entities, it’s much less of an option. You could try also to create the conditions programmatically with the API depending on what data you are querying against.
Again, I know this is not ideal, but please know that we are marshaling resources against this as best we can right now.
Thanks @hross. I appreciate the transparency! If we can help in any way, please let me know. We’re glad to be as useful as we can to get this resolved.
And, yeah, Plan B is a no-go for us. Too many alerts!
I think there could be mileage in the automated method. You could write your NRQL query with the FACET and extract that into a CSV. Use the CSV as your configuration to create each alert under an alert policy by script.
If you also write the removal script, you reverse once the FACET issue is resolved.
So we’ve actually decided to just trust our users more That doesn’t mean that we don’t want this feature fixed so that we can FACET in alerting, but we’re just going to give our teams templates that they can copy into their own policies and leave them to create their own conditions.
It’s a work-around, not a fix — but at least we’ll have a chance of hitting our deadline.
LOL – that’s kind of the same response I gave our senior management. “Uhh, good luck with that trust thing!”
We actually decided that we are going to use the API to back up the policies, conditions, and notification channels just in case someone makes a mistake.
Trust, but verify is a great operational motto!
And I want to clarify because I don’t want to come off as snide. We should 100% trust our people to be great decision makers. We hire and work with smarties! That said, building repetitive alerts policies manually is is thing that ALL humans will mess up because… HUMAN. So I like your approach @jbiggley
ROFL – oh no, I get it. Trust is a hard thing. I like to live by the mantra “When people try and show you who they are, believe them”. If folks (including engineers!) want to act like they can’t be trusted, well then OK!
“I trust everyone. I just don’t trust the devil inside them.”
― Troy Kennedy Martin
I think we actually have lift off. The product team should have resolved this issue (and it looks good to us). It would be great to know if it is working for you all!
Explorers Hub Treasure Hunt
Can confirm. We just tested and incidents were created for each entity using a NRQL query. Nicely done @hross and product team!
I definitely deserve all the credit here, so I’m glad you noticed! Very happy that it is working for you so that you can Go Live tomorrow with all your features in place!
To clarify, you may optionally include one FACET in a NRQL alert condition but no more than one?
That’s my understanding
That is correct. NRQL alerts permit only one FACET whereas NRQL queries in Insights can have multiple FACETs.