US2 X.509 Certificate Service is presenting errors
Incident Report for SecureAuth Service
Postmortem

Events on Saturday 22nd:

15:30 - Report by Andrew Wood
Affected : Certs & NGE

15:40 - Started review of the CA's
16:30 - Decided to Restart all CA's @US2

17:30 - Issue resolved

Root Cause:

Downstream of PROD-449: Upgrade of Vmware Tools on Us2 Servers. Vendor updated VMWARE tools on servers but since installer didn't request reboot for CA's, they were not rebooted.

As result, CA's behavior was unstable.

Resolution:

After restarting all the CA's at US2, one by one, the service was restored.

Included in procedure used by provider, to reboot the servers after vmware tools upgrade, as a best practice recommended by VMWARE.

Posted Feb 26, 2020 - 15:02 UTC

Resolved
This incident has been resolved.
Posted Feb 22, 2020 - 22:01 UTC
Update
We are continuing to monitor for any further issues.
Posted Feb 22, 2020 - 17:00 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 22, 2020 - 16:30 UTC
Investigating
We are currently investigating this issue.
Posted Feb 22, 2020 - 15:30 UTC
This incident affected: SecureAuth Cloud Services (X.509 Certificate Service (SHA2) - US2).