Increased API Errors
Incident Report for SecureAuth Service
Postmortem

Incident Description:

Customers attempting to log into the Cloud IdP platform were experiencing 504s.

Customer Impact:

A subset of Cloud IdP Customers

Root Cause:

Between 4/28/2022 21:00 UTC – 4/29/2022 01:00 UTC; as part of a routing update to backend services, a number of Broker services used by the Cloud IdP platform failed to update to the necessary version which did not support other changes being applied.

Resolution:

All failing Broker services were identified and the appropriate version was applied shortly after.

Corrective Actions:

During the upgrade process a missing flag was not detected by the script and failed to appropriate upgrade all backend services to the correct version. An audit of the update scripts is being performed to ensure important missing flags or settings fail the upgrade and allow the operator to take the appropriate corrective actions prior to the upgrade commencing.

Posted Apr 29, 2022 - 19:49 PDT

Resolved
This incident has been resolved.
Posted Apr 29, 2022 - 13:54 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Apr 29, 2022 - 02:35 PDT
Investigating
We are observing an increased number of API Errors with our Cloud IdP Customers. Users may experience issues logging in. We are investigating the issue and will post an update once available.
Posted Apr 28, 2022 - 23:57 PDT