Issue with One-time Passcodes
Incident Report for SecureAuth Service

Incident Description

On Wednesday, July 10th at approximately 1:10PM UTC, our monitoring infrastructure detected potential SMS issues in our West Coast data center. Investigations determined the cause to be related to internal database complications. The issue was resolved at approximately 1:50PM UTC.

Root Cause

A database sync error was discovered to be caused by an internal back-office software release. As changes were still being made to both databases, overtime this sync error led to data integrity issues, which prevented some SMS transactions from being carried out.

Corrective Actions

The immediate problem was resolved by manually copying over staged records in the sync to the destination database and re-establishing a new sync connection between the two databases.

The following measures are being taken to prevent an incident of this type from happening in the future:

" Improved reporting, alerting, and response around database sync issues

" Additional database constraints added for mitigation

Posted Jul 16, 2019 - 00:16 UTC

This incident has been resolved. RCA to follow within 24 hrs.
Posted Jul 10, 2019 - 14:51 UTC
The issue has been identified and a fix has been put into place. We are continuing to monitor and test intensively.
Posted Jul 10, 2019 - 14:08 UTC
We are currently investigating an issue with One-time passcodes being delivered via SMS, Voice, and Push. We will update as we know more.
Posted Jul 10, 2019 - 13:42 UTC
This incident affected: Telephony Service (SecureAuth US1 Datacenter, SecureAuth US2 Datacenter, Nexmo Voice API), SMS Service (SecureAuth US1 Datacenter, SecureAuth US2 Datacenter, Telephony Provider SMS API, 3rd Party Mobile Carrier), and Push-to-Accept (Login Requests)/Push Notification Service (SecureAuth US1 Datacenter, SecureAuth US2 Datacenter).