Incident Description
At approximately 06:15 UTC Saturday, February 2, 2019 our primary data center suffered an outage that affected the delivery of One-Time Passcodes. At 06:29 UTC, we performed a failover to our other data center. Although functionality was restored by 06:51 Saturday, February 2, the system remained re-directed until; 14:30 UTC Saturday, February 2, 2019 as a precautionary measure. At approximately 14:30 UTC Saturday, Feb 2nd service was restored and normal operation of the primary data center resumed.
Root Cause
The root cause was determined to be an inadvertent reboot of servers at our data center hosting provider. Human error in the configuration of an automated patching process managed by our hosting provider caused the reboot of a large number of systems at the hosting provider including those supporting SecureAuth.
Corrective Actions
Our hosting provider are currently conducting their own Root Cause Analysis. This report will be updated with those details when available.
In response to these events, SecureAuth is updating our standard procedures to include the use of additional methods for traffic redirection in addition to our current methods of failover. This will reduce the amount of time before customer traffic begins reaching the secondary data center.