RCA – SecureAuth Authenticate iOS Release

Problem Description:

On September 19, 2024 at 10:00AM PDT, SecureAuth released SecureAuth Authenticate version 25.0.18 to the IOS App Store. The application was targeted to 1% of the userbase with the previous version installed; however, users could still download the new version manually if desired. At approximately 11:00AM PDT, we receive internal and external reports that version 25.0.18 of SecureAuth Authenticate has broken preexisting enrollments, and users were unable to re-enroll to resolve the issue.

Cause:

Within the update process for SecureAuth Authenticate version 25.0.18, a format conversion of account data to a new arc-6 architecture is performed. The format conversion process failed due to missing group entitlements within the project configuration. This resulted in the SecureAuth Authenticate App’s inability to properly load account information, thus breaking functionality for end users. While no account data was lost, previous enrollments were inaccessible, and users were prompted to enroll, but could not. These entitlements were not discovered during QA due to a caching of configurations inside of Apple’s TestFlight system where the application was being tested. The only way to clear this cache is a factory reset of the device. Unknown to developers, these cached configurations held onto the missing group entitlements that were not present in the GA version of the application that was released.

Recovery:

The engineering team initiated resolution efforts on two fronts: Front 1: Revert to the previous version as an initial interim mitigation to restore functionality to impacted users as quickly as possible while a separate team focused on providing permanent fix for the following release. However, upon revert efforts, it was determined that this option was not viable due to technical complications brought on by the update and Apple compliance standards related to the Apple Watch application. Focus of efforts quickly shifted to Front 2. Front 2: Identify the cause of the format conversion failure and implement the fix. Upon successful completion of QA validation, submit to the iOS App Store for urgent review and publishment.

Resolution: The format conversion process failed due to missing group entitlements within the project configuration. The fix reapplied the necessary group entitlements, restoring the app’s ability access the account data.

Timeline:

Sep 19, 2024

• 10:00 AM PDT - SecureAuth Authenticate version 25.0.18 released in iOS App Store
• 11:00 AM PDT – Internal teams discover the issue with App release and Engineering Teams are notified
• 11:09 AM PDT – Incident bridge started and Engineering teams begin investigating the issue
• 11:15 AM PDT – Engineering Teams begin efforts to revert App to previous version in iOS App Store.
• 11:20AM PDT – Status Page updated to inform customers to hold from application updates until further notice
• 11:20 AM PDT – Engineering Teams continue to investigate root cause of the issue while also working on reverting to previous version on the iOS App Store
• 12:30 PM PDT – Confirmed that the issue was not isolated to iOS 18
• 1:00 PM PDT – Discarded rollout option due to complications with Apple compliance standards for Wearable App
• 1:00 PM PDT – Engineering teams refocus to provide patch for the Authenticate app.
• 2:05 PM PDT – Cause of the issue identified as a failure of the format conversion process due to missing group entitlements within project configuration
• 2:25 PM PDT – Fix is implemented and QA validation is initiated
• 3:00 PM PDT – New build published for Urgent Review to App Store.
• 4:12 PM - New version is deployed. Impacted users were notified to download the version 25.1.18 with preexisting enrollments intact.
Support and Engineering teams continue to monitor the situation closely with customers.

Corrective Actions:
• Work with Apple to review their TestFlight requirements and determine why configurations were being cached, discover the standard duration of the cache period, and identify the steps needed to ensure the cache is cleared and updated configurations are being used during TestFlight QA processes.
• Improve the current Pull Request and Code Review process in Mobile Development in order to mitigate the impacts of missing configurations and improve code release standards.
• Add test cases to our QA suite to cover fresh devices, as it was determined that if wiped or “new” devices were being used for testing, the cached configurations would have been discovered.