SaaS-IDP Connection Issue.
Incident Report for SecureAuth Service

Incident Description

At 4/28/2023 3:15 AM UTC, the DevOps team did a planned, minor update to traffic management appliances during a maintenance window. While no downtime was expected and all configurations were tested in lower environments, the new update in production did not respect TLS cipher suite configuration as expected. This created a TLS negotiation issue between the SA IdP appliances and the traffic management appliances, blocking some logins from occurring.

Unfortunately, the post-change test suite did not detect the issue. After hearing about difficulties from some customer reports and first-hand observations, the DevOps team identified the issue and rolled back the change to restore service at 4:15 AM UTC (4/28/23).

Root Cause

An update to the update to traffic management appliances did not obey a declarative configuration for TLS cipher suites. This caused a TLS negotiation failure between SA IdP appliances and the traffic management appliances, blocking some logins.

Corrective Actions

We apologize for any inconvenience caused during this disruption in service. The SecureAuth team will pause any further changes to the traffic management appliances until we have sorted out the associated configuration issue (which we plan to solve in the next few weeks). In addition, any future updates done to these appliances will be done during a planned maintenance window where we notify customers that there may be downtime expected (regardless of such risk). Finally, we will review our post-change test plan to evaluate adding tests that would discover such trouble points in the future.

Posted Apr 28, 2023 - 14:28 PDT

We have found the issue and have made the proper steps to restore service for affected for SaaS-IdP tenants. Further details will be provided in an RCA once the operations team concludes the issue investigation.
Posted Apr 27, 2023 - 21:15 PDT
We are experiencing technical difficulties during our maintenance window with all SaaS-IdP connections.
Posted Apr 27, 2023 - 20:15 PDT
This incident affected: SaaS/Full Cloud Components (SaaS/Full Cloud Identity Platform, SecureAuth Connector).