After September update from Channels to Workflow, new alerts are not triggered

We are having issues when creating new alert policies, seems related to the update made on september 1st regarding to notifications to workflow, the situtation is eventhough the alert is correctly created and the warning and critical threasholds in the configuration shows that must be openned new incidents, the policies does not trigger this incidents after save them, they are correctly activated, they are not bein visualizated in the overview / issues incidents dashaboards neither. To clarify this only happens with new policies, the policies that were created before augusts 29th does not have problems so far, they are triggering the Incidents and notifications correctly.

But there is not any incident triggered by this policies:

We appreciate your urgent help our accountId is : 3016614

Hi @devopsmonitoreo

Thanks for reaching out, I hope you are well.

Unfortunately I am unsure as to why this issue is occurring. Please note I will loop in a supporting engineer to have a look.

Can you provide a permalink to where you are noting this issue occur. Only Relic will be able to revise the link.

Also Can you provide a permalink to a new alert that isnt functioning as expected.

Hi, dcody, thanks for the response, this alerts doesn’t have permalink, but I can share a video explaining the issue, and the names of the rules that are failing:

Here is the video explaining the issue: https://screencast-o-matic.com/watch/c3QDozVOuGG

You can reproduce an check the new alerts created since September 1st that are not working

EncolamientoBBBrecording-Colombia

Encolamiento_Correos_SENAv1

Encolamiento_correos_SENA

Hi @devopsmonitoreo

Thanks for the update here, and for the screencast it really gives a clear insight into the issue.

I have gone ahead and created a case on your behalf with the alerts team, they will reach out via email soon. I have also mentioned to the team to expedite this as the issue has been present since September 1st.

Please do let me know should you need any additional support.

@devopsmonitoreo - Can you verify that you are still not seeing any incidents. I was able to see some as recent as 3 hours ago as indicated in the screenshot. Please note that by default only active alert incidents will show in the Issues & Activity. By clearing the state = 'active' filter in the search box you will be able to see everything both open and closed.

1 Like

Hi Dkoyano, if you check my previous video, we are having issues with new alerts created after september 1st, that alerts you show in the dashboard are created before september 1st, but for example this one “Encolamiento_Correos_SENAv1” that we show also in the vide have several incidentes every day, but none one is triggered:

Alert policiy: Encolamiento_Correos_SENAv1

Condition: ENCOLAMIENTO_CORREOS_SENA_SUPERIOR_25000

Hi @devopsmonitoreo

I’ve checked what is being evaluated by some of your conditions like Encolamiento_Correos_SENA and I could see that the value returned by the query is a string instead of a numeric value.

Apparently, the attribute queue_cont is a type string and with this, the conditions can’t evaluate the changes in the results, not creating or closing the incidents.

If you check the results of this query:

FROM NrAiSignal SELECT * where conditionID = <YOUR_CONDITION_ID_HERE>

You are going to see in the error attribute showing Non numeric (string) value cannot be evaluated..


To fix this, I’d suggest including the numeric function in your query. Eg:

SELECT latest(numeric(queue_count)) from Log where hostname ='senaSofiaBD' and client='SENA' and database ='territorio2_0'

This function will convert the string values into numeric values and with this the conditions will be able to evaluate the information.

Please let me know if you have any doubts.


Also if this helped resolve the issue please consider selecting the :white_check_mark: Solution option below this post. Find more information here: How to Use the Official Solution Checkbox


thanks

Rodrigo

Thanks!!, that was the issue, for some reason the new alerts fails for that conversion, we have applied the workarround and they start working!, thanks for your great help.

@devopsmonitoreo

Im happy to hear this was resolved for you!

Wishing you a great day.