Interruption of Service in Eastern Data Center
Incident Report for SecureAuth Service
Postmortem

Incident Description

At approximately 22:12 UTC internal monitoring detected a disruption in service from our East Coast data center. Investigation determined it to be related to internal database issues. All traffic was re-0directed to our west coast data center. The issue was resolved at approximately 23:15 UTC.

Root Cause

A connectivity failure between the east and west coast data centers disrupted database replication. This disruption caused a loss of appliance licensing data, which resulted in some appliances being treated as unlicensed, which impaired all functions of the impacted appliances.

Corrective Actions

The immediate problem was resolved by a database restore and manual data merge. Subsequently, the communications failure was identified as a mismatch between load balancer and firewall rules, which has been corrected. Our active test and monitor tools are being extended to monitor the load balancer - firewall communication channel to provide earlier detection and quicker identification of this class of issue.

Posted Jul 12, 2019 - 21:00 UTC

Resolved
A fix has been implemented and tested. Traffic has been restored to the east coast data center and all services should be operating normally.
Posted Jul 11, 2019 - 04:09 UTC
Identified
At approximately 22:12 UTC internal monitoring detected a disruption in service from our East Coast data center. Investigation determined it to be related to internal database issues. We have re-directed all traffic to our west coast data center while a fix is being implemented. All services should be functioning normally. We will update this page as the situation evolves.
Posted Jul 11, 2019 - 01:56 UTC
This incident affected: Push-to-Accept (Login Requests)/Push Notification Service (SecureAuth US1 Datacenter), Telephony Extension/DTMF Service (SecureAuth US1 Datacenter), Telephony Service (SecureAuth US1 Datacenter), and SMS Service (SecureAuth US1 Datacenter).