Incident Description

At approximately 0450 UTC on Sunday, June 23, 2019 we became aware of authentication difficulties for customers in the eastern ares of the U.S. We performed an internal failover and service was restored by 0506 UTC.

Root Cause

The root cause was traced to an issue with a storage fabric upgrade being performed by one of our data center vendors. This issue cause both out primary and backup database servers in that data center to go offline. We performed a DNS redirection to send traffic bound for that data center to an alternate center in our cloud.

Corrective Actions

We are working with our data center hosting company to understand the root cause of the storage fabric outage. We are investigating ways to further separate our primary and backup servers and are creating tailored monitoring the speed identification and mitigation of any similar issues in the future.