Washington D.C. constantly showing downtime on up server

I’m seeing issues with the public minions in D.C. where it’s showing outages of 10 to 45 minutes of downtime about 4 to 5 times a day. I tend to see the issues at 0000-0200, 1200-1400, and 1900 to 2100 Eastern, but there are outliers. The issue began at the same time that the Atlanta minions were dropped. I have confirmed that all the minions in the current list (checked 3 times a week, including about 30 minutes ago) against IPTABLES and the ASA, and they are whitelisted. I have also confirmed uptime during these times through PING, NC, and load testing the server via LoaderIO to confirm uptime. I also have a remote load balancer with CloudFlare monitoring the same health page 3x as frequently, and it is not reporting any issues, and hasn’t dropped the servers from rotation at all.

My questions are as follows: Are there any known issues in D.C.? Can you confirm that all minions are able to route to Atlanta appropriately? Is there a way to determine which minions are failing to connect? And, finally, can we confirm that the current list is up to date?

Based on the behavior, it seems like either there is an undocumented minion that I haven’t whitelisted that the ASA is dropping, or a minion is failing or oversubscribed. Please let me know. I’m happy to share IP’s, logs and show runs to an NREL employee in a private message. Thanks,

Mike Burroughs

Hi @mike13 -

Sorry to hear about the trouble.

Although we have not recently experienced any issues any issues with the performance of our DC monitor, we did recently update our IP addresses. The announcement can be found here:

The list of new IP addresses can be found here:


Apologies if this was already on your radar, but I wanted to make sure we got this possibility out of the way.


Turns out, rather unsurprisingly, I screwed this one up. As it turns out, and are different IP’s, but my fat fingers did not realize this. Thanks for the help.

It’s cool @mike13 - we ALL do this sort of thing. Never ask me to make an appointment on a calendar, for example. I will probably book it for the wrong year. Good luck and let us know if you need anything else!