Interruption of Service in Eastern Data Center

Incident Report for SecureAuth Service

Postmortem

Incident Description

At approximately 22:12 UTC internal monitoring detected a disruption in service from our East Coast data center. Investigation determined it to be related to internal database issues. All traffic was re-0directed to our west coast data center. The issue was resolved at approximately 23:15 UTC.

Root Cause

A connectivity failure between the east and west coast data centers disrupted database replication. This disruption caused a loss of appliance licensing data, which resulted in some appliances being treated as unlicensed, which impaired all functions of the impacted appliances.

Corrective Actions

The immediate problem was resolved by a database restore and manual data merge. Subsequently, the communications failure was identified as a mismatch between load balancer and firewall rules, which has been corrected. Our active test and monitor tools are being extended to monitor the load balancer - firewall communication channel to provide earlier detection and quicker identification of this class of issue.

Posted Jul 12, 2019 - 14:00 PDT

Resolved

A fix has been implemented and tested. Traffic has been restored to the east coast data center and all services should be operating normally.

Posted Jul 10, 2019 - 21:09 PDT

Identified

At approximately 22:12 UTC internal monitoring detected a disruption in service from our East Coast data center. Investigation determined it to be related to internal database issues. We have re-directed all traffic to our west coast data center while a fix is being implemented. All services should be functioning normally. We will update this page as the situation evolves.

Posted Jul 10, 2019 - 18:56 PDT

This incident affected: SecureAuth Cloud Services (Push-to-Accept Service - US1, SMS Service - US1, Telephony Extension/DTMF Service - US1, Telephony Service - US1).