Incident Description
At approximately 06:00 UTC Sunday, February 3, 2019 our primary data center suffered an outage related to load-balancers supporting SecureAuth traffic. The outage was not identified until approximately 09:55 UTC Sunday, February 3, 2019 due to the nature of the outage. Restorative work was completed by approximately 10:05 UTC and normal operation resumed.
Root Cause
In the process of addressing issues caused by the February 2nd inadvertent server reboot, our data center hosting provider caused a second unanticipated reboot of some of our supporting servers. In addition, we discovered a gap in SecureAuth’s internal monitoring, alerting and escalation technologies which resulted in the issue being prolonged for an unacceptable length of time.
Corrective Actions
Our hosting provider is currently conducting their own Root Cause Analysis. This report will be updated with those details when available.
SecureAuth has identified and corrected the internal technology issue that delayed alerting and response. Additional monitors are under development that will utilize other indicators not currently leveraged to better inform our monitoring and alerting capabilities. Automated daily testing of the alerting and escalation system has been established.