Issue with One-time Passcodes
Incident Report for SecureAuth Service
Postmortem

Incident Description

On Wednesday, July 10th at approximately 1:10PM UTC, our monitoring infrastructure detected potential SMS issues in our West Coast data center. Investigations determined the cause to be related to internal database complications. The issue was resolved at approximately 1:50PM UTC.

Root Cause

A database sync error was discovered to be caused by an internal back-office software release. As changes were still being made to both databases, overtime this sync error led to data integrity issues, which prevented some SMS transactions from being carried out.

Corrective Actions

The immediate problem was resolved by manually copying over staged records in the sync to the destination database and re-establishing a new sync connection between the two databases.

The following measures are being taken to prevent an incident of this type from happening in the future:

" Improved reporting, alerting, and response around database sync issues

" Additional database constraints added for mitigation

Posted 8 days ago. Jul 16, 2019 - 00:16 UTC

Resolved
This incident has been resolved. RCA to follow within 24 hrs.
Posted 14 days ago. Jul 10, 2019 - 14:51 UTC
Monitoring
The issue has been identified and a fix has been put into place. We are continuing to monitor and test intensively.
Posted 14 days ago. Jul 10, 2019 - 14:08 UTC
Investigating
We are currently investigating an issue with One-time passcodes being delivered via SMS, Voice, and Push. We will update as we know more.
Posted 14 days ago. Jul 10, 2019 - 13:42 UTC
This incident affected: Telephony Service (SecureAuth US1 Datacenter, SecureAuth US2 Datacenter, Nexmo Voice API), SMS Service (SecureAuth US1 Datacenter, SecureAuth US2 Datacenter, Telephony Provider SMS API, 3rd Party Mobile Carrier), and Push-to-Accept (Login Requests)/Push Notification Service (SecureAuth US1 Datacenter, SecureAuth US2 Datacenter).