Issue with One-time Passcodes

Incident Report for SecureAuth Service

Postmortem

Incident Description

On Wednesday, July 10th at approximately 1:10PM UTC, our monitoring infrastructure detected potential SMS issues in our West Coast data center. Investigations determined the cause to be related to internal database complications. The issue was resolved at approximately 1:50PM UTC.

Root Cause

A database sync error was discovered to be caused by an internal back-office software release. As changes were still being made to both databases, overtime this sync error led to data integrity issues, which prevented some SMS transactions from being carried out.

Corrective Actions

The immediate problem was resolved by manually copying over staged records in the sync to the destination database and re-establishing a new sync connection between the two databases.

The following measures are being taken to prevent an incident of this type from happening in the future:

" Improved reporting, alerting, and response around database sync issues

" Additional database constraints added for mitigation

Posted Jul 15, 2019 - 17:16 PDT

Resolved

This incident has been resolved. RCA to follow within 24 hrs.

Posted Jul 10, 2019 - 07:51 PDT

Monitoring

The issue has been identified and a fix has been put into place. We are continuing to monitor and test intensively.

Posted Jul 10, 2019 - 07:08 PDT

Investigating

We are currently investigating an issue with One-time passcodes being delivered via SMS, Voice, and Push. We will update as we know more.

Posted Jul 10, 2019 - 06:42 PDT

This incident affected: SecureAuth Cloud Services (3rd Party Mobile Carrier, Nexmo Voice API, Push-to-Accept Service - US1, Push-to-Accept Service - US2, SMS Service - US1, SMS Service - US2, Telephony Provider SMS API, Telephony Service - US1, Telephony Service - US2).