At approximately 2:30 PM PST on 6/29/2018 we began to receive reports of certain individual users either not receiving 2FA authentication prompts or receiving errors during the login process. The issue was traced to the Fraud Prevention Service, and the cause identified by approximately 3:45 PM PST. Service was restored to all users by approximately 4:15 PM PST.
After a thorough analysis and review of our systems we have determined that the problem originated from a change in the format of data returned by one of our service providers. Specifically, one block of data that used to be provided was no longer being returned. This induced an error condition in the IdP and prevented certain users from logging in.
After working with our service provider, it was determined that one of their suppliers had changed the licensing terms for the data in question. This rendered certain data elements unavailable through the standard API.
The service provider supports a “basic” and “advanced” version of their API. The “basic” API was the one in use at the time of the issue. The “advanced” API did not suffer the change noted above. The Fraud Prevention Service configuration was changed to point at the “advanced” API, which resolved the issue. An internal issue report has been logged requesting that the Fraud Prevention Service check for and synthesize any missing portions of the data prior to returning them to the IdP. We have engaged with the service provider to address the issue in the basic API. Until we have confirmation that the data will be available in the standard API, we will continue to use the advanced API. Our service provider is actively seeking alternate suppliers for the data that is missing.