Data Center Maintenance

Impacted Customers: A subset of customers using US2/West SecureAuth Datacenter

Impacted Services: All cloud services

Incident Date: Dec. 14, 2020

Incident Description:

Because of vendor initiated network related maintenance, SecureAuth failed over all cloud services from our US2 to US1 datacenter via a DNS failover. 

During the failover process a human error was made impacting a specific DNS record that affected Push-to-Accept transactions.  This issue was limited to customers systems that did not recognize the DNS change, and remained on our US2 datacenter.  This issue was identified and resolved Dec 15th 7:55 AM PST. 

Root Cause:

• The root cause for customer networks and IdP appliances not respecting the DNS change is still under investigation. 
• The DNS change was due to human error. 

Corrective Actions:

• Further investigation is required to understand why some customer networks did not respect the DNS change.
• The DNS error will be avoided by updating the DNS failover playbook and checklist