P2A - Push to accept
Incident Report for SecureAuth Service
Postmortem

Impacted Customers: All customers using Push and Link Services

Impacted Services: All cloud services

Incident Date: June 6, 2020

Incident Description:

A certificate related issue impacted SecureAuth cloud services. The scope of impact was limited due to the nature of how CRL updates are handled by Windows systems. Because of how mobile platforms handle CRL updates, Push and link services were impacted first.

Timeline of events:

  • - At approximately 21:20 UTC on 06/06/2020 SecureAuth monitoring alerted on transaction service issues (considered minor/non-customer impacting)
  • - At approximately 22:50 UTC, customers report Push issues
  • - At approximately 23:55 UTC, root cause of issue was confirmed
  • - At approximately 00:15 UTC 06/07/2020, CA vendor was contacted
  • - At approximately 03:03 UTC, CA vendor reissued certificate
  • - At approximately 03:29 UTC, some critical services were restored
  • - At approximately 05:00 UTC, all services were restored

Root Cause:

The secureauth.com certificate was revoked by our CA vendor due to an administrative error on their part.

The secureauth.com certificate was recently reissued by our CA vendor and updated across all SecureAuth services. During the issuance process, the vendor made an administrative error that resulted in the certificate being revoked by their systems on 06/06/2020.

SecureAuth received no notification from the vendor.

---Verbatim text of vendor RCA---

Our system have issued the certificate with he expired documents for the Deluxe OV/EV signing certificate without proper verification.

Our admins says, When you provide documentation during your initial certificate request and validation for your Deluxe certificate's, those documents can be re-used for future requests for up to 825 days.

The processes we had in place to check those dates failed and resulted in certificates being issued using expired documents. That issue has now been fixed, but for this subset of customers we needed to take action by revoking it.

But again We have already initiated a re-key request in the your's account on you behalf, validated your latest documents and issued the certificate.

This has never happened from our end before.

And it's a known issue kind of thing which was happened by our system.

We apologize for the inconvenience caused to you.

---End of verbatim text of vendor RCA---

Corrective Actions:

  • - CRL monitoring alerts will be implemented
  • - Add monitoring that leverages actual mobile device to detect mobile service provider related issues
  • - Reconsider CA vendor choice (currently GoDaddy)
Posted Jun 10, 2020 - 09:44 PDT

Resolved
This incident has been resolved.
Posted Jun 06, 2020 - 20:48 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 06, 2020 - 20:31 PDT
Identified
We identified the problem to be related to our certificate provider (GoDaddy). We will be working on a resolution once the issue has been resolved at GoDaddy.
Posted Jun 06, 2020 - 18:27 PDT
Investigating
We are having problems with P2A functionality. We are working on it.
Posted Jun 06, 2020 - 17:14 PDT
This incident affected: SecureAuth Cloud Services (Push-to-Accept Service - US1, Push-to-Accept Service - US2).