SMS and Telephony outage
Incident Report for SecureAuth Service
Postmortem

Incident Description

Starting around 1:12 PM on June 8, our monitors detected an issue with our primary, third-party SMS (simple message service) provider used for sending MFA (multi-factor authentication) codes. This issue caused intermittent delays when sending messages. While we have a secondary provider, the automatic failover did not occur as expected due to the nature of the primary provider’s SMS issue: the primary actually acknowledged our system’s call to send a message and would queue it but ultimately fail to deliver it promptly. Once our team detected the issue via our monitoring system and performed an investigation, we initiated a manual failover to our secondary SMS provider to reduce any service disruptions for our customers around 2:34 PM. Our primary provider resolved their issue at 5:06 PM, and we manually switched back around 8:43 PM.

Root Cause

One of our 3rd party service providers experienced an intermittent service disruption which impacted MFA SMS delivery.

Follow Up Actions

We apologize for any disruption in services during this incident. We will review our SMS provider failover thresholds to make them more sensitive to prevent this type of challenge from affecting our customers again.

Reference

Telesign Status Page Incident: https://status.telesign.com/incidents/t201ymq6kl1f

All dates/times are UTC.

Posted Jun 09, 2023 - 16:27 PDT

Resolved
We have switched back to our primary provider and all services are operational
Posted Jun 08, 2023 - 20:23 PDT
Update
We are continuing to monitor for any further issues.
Posted Jun 08, 2023 - 07:58 PDT
Update
We are continuing to monitor for any further issues.
Posted Jun 08, 2023 - 07:41 PDT
Monitoring
Switching to the secondary provider for SMS appears to have resolved the issue for now. We will continue to monitor.
Posted Jun 08, 2023 - 07:39 PDT
Identified
The issue has been identified as a fault out the telephony provider side so we are switching to a backup provider.
Posted Jun 08, 2023 - 07:36 PDT
Investigating
Our Telephony provider is having an issue with SMS and Telephony.
Posted Jun 08, 2023 - 07:29 PDT