Investigating Potential Issues with Fraud Prevention Service
Incident Report for SecureAuth Service
Postmortem
Incident Description

At approximately 2:30 PM PST on 6/29/2018 we began to receive reports of certain individual users either not receiving 2FA authentication prompts or receiving errors during the login process. The issue was traced to the Fraud Prevention Service, and the cause identified by approximately 3:45 PM PST. Service was restored to all users by approximately 4:15 PM PST.

Root Cause

After a thorough analysis and review of our systems we have determined that the problem originated from a change in the format of data returned by one of our service providers. Specifically, one block of data that used to be provided was no longer being returned. This induced an error condition in the IdP and prevented certain users from logging in.

After working with our service provider, it was determined that one of their suppliers had changed the licensing terms for the data in question. This rendered certain data elements unavailable through the standard API.

Corrective Actions

The service provider supports a “basic” and “advanced” version of their API. The “basic” API was the one in use at the time of the issue. The “advanced” API did not suffer the change noted above. The Fraud Prevention Service configuration was changed to point at the “advanced” API, which resolved the issue. An internal issue report has been logged requesting that the Fraud Prevention Service check for and synthesize any missing portions of the data prior to returning them to the IdP. We have engaged with the service provider to address the issue in the basic API. Until we have confirmation that the data will be available in the standard API, we will continue to use the advanced API. Our service provider is actively seeking alternate suppliers for the data that is missing.

Posted 4 months ago. Jul 02, 2018 - 23:12 UTC

Resolved
The issue has been resolved and the RCA will be provided on Monday.
Posted 4 months ago. Jun 29, 2018 - 22:15 UTC
Monitoring
Our cloud engineering team has implemented a fix and we are continuing to monitor the situation.
Posted 4 months ago. Jun 29, 2018 - 21:43 UTC
Update
We are continuing to investigate this issue.
Posted 4 months ago. Jun 29, 2018 - 21:07 UTC
Investigating
We are currently investigating potential issues with our IdP Fraud Prevention services. We will provide an update as soon as we have more information.
Posted 4 months ago. Jun 29, 2018 - 21:07 UTC
This incident affected: SMS Service (SecureAuth US East Datacenter, SecureAuth US West Datacenter, Nexmo SMS API), Telephony Service (SecureAuth US East Datacenter, SecureAuth US West Datacenter, Nexmo Voice API), and Telephony Extension/DTMF Service (SecureAuth US East Datacenter, SecureAuth US West Datacenter).