US2 X.509 Certificate Service is presenting errors

Incident Report for SecureAuth Service

Postmortem

Events on Saturday 22nd:

15:30 - Report by Andrew Wood
Affected : Certs & NGE

15:40 - Started review of the CA's
16:30 - Decided to Restart all CA's @US2

17:30 - Issue resolved

Root Cause:

Downstream of PROD-449: Upgrade of Vmware Tools on Us2 Servers. Vendor updated VMWARE tools on servers but since installer didn't request reboot for CA's, they were not rebooted.

As result, CA's behavior was unstable.

Resolution:

After restarting all the CA's at US2, one by one, the service was restored.

Included in procedure used by provider, to reboot the servers after vmware tools upgrade, as a best practice recommended by VMWARE.

Posted Feb 26, 2020 - 07:02 PST

Resolved

This incident has been resolved.
Posted Feb 22, 2020 - 14:01 PST

Update

We are continuing to monitor for any further issues.
Posted Feb 22, 2020 - 09:00 PST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Feb 22, 2020 - 08:30 PST

Investigating

We are currently investigating this issue.
Posted Feb 22, 2020 - 07:30 PST