High Disk Utilization Alarm Not Closing (or Opening) Even Though Conditions Warrant It

Recently I have noticed that one of our high disk utilization alarms closed later than it should have based on the criteria. And the e-mail stating that the alarm was closed was going out over an hour later than expected. If a person is just looking at e-mails, it gives the impression that the alarm is open longer than it is actually open.

Original Settings
readUtilizationPercent = 100 for at least 60 mins

Example
12:00 to 1:30 AM - Disk Util = 100% - Expectation = alarm opens at 1:00 AM - What Actually Happens = alarms opens at 1:00 AM
1:30 AM to 1:45 AM - Disk Util < 20% - Expectation = alarm stays open - What Actually Happens = alarm stays open
1:45 AM to 2:45 AM - Disk Util < 20% - Expectation = alarm stays open because 60 min timer was reset - What Actually Happens = alarm stays open
2:45 AM - Disk Util < 20% - Expectation = alarm closes, e-mail is sent at 2:45 AM - What Actually Happens - alarm stays open until 3 AM, email is sent at 4 AM

On Friday Oct 22 at 7 AM, I changed this alarm to:
readUtilizationPercent = 99 for at least 60 mins

This had the effect of shutting off the alarm altogether. Even though conditions warrant an alarm, no alarm is raised.
image

On Tue Oct 26 at 7 AM, I changed this alarm to:
readUtilizationPercent = 98 for at least 60 mins

There was no change to the behavior. Even though conditions warrant an alarm, no alarm is raised.
image

On Wed 27 at 7 AM, I set the alarm back to 100%. My expectation is that it will return to the pre-Oct 22 behavior. I will report back with the results. However, if we are back to the original problem described above, how do we fix that?

Expected Behavior: 60 mins after the disk util drops below 100% util, close the alarm and send an e-mail.

Oct 28 2021 UPDATE
Yesterday I set the threshold back to 100% and today the alarm opened as expected and closed as expected. However, the e-mail was created 60 min after the alarm closed. Is there anyway that the e-mail can be created and sent right after the alarm closes?


image

This is from the e-mail that is sent. We are GMT -500 so here the opening time is right (1 AM) but the closing time is wrong (3:48 AM), it should be 2:48 AM.
image

Hello, Thanks for the updates. To get the information I need to look into this I’ve responded through the internal ticket.