At 4/28/2023 3:15 AM UTC, the DevOps team did a planned, minor update to traffic management appliances during a maintenance window. While no downtime was expected and all configurations were tested in lower environments, the new update in production did not respect TLS cipher suite configuration as expected. This created a TLS negotiation issue between the SA IdP appliances and the traffic management appliances, blocking some logins from occurring.
Unfortunately, the post-change test suite did not detect the issue. After hearing about difficulties from some customer reports and first-hand observations, the DevOps team identified the issue and rolled back the change to restore service at 4:15 AM UTC (4/28/23).
An update to the update to traffic management appliances did not obey a declarative configuration for TLS cipher suites. This caused a TLS negotiation failure between SA IdP appliances and the traffic management appliances, blocking some logins.
We apologize for any inconvenience caused during this disruption in service. The SecureAuth team will pause any further changes to the traffic management appliances until we have sorted out the associated configuration issue (which we plan to solve in the next few weeks). In addition, any future updates done to these appliances will be done during a planned maintenance window where we notify customers that there may be downtime expected (regardless of such risk). Finally, we will review our post-change test plan to evaluate adding tests that would discover such trouble points in the future.