Events on Saturday 22nd:

15:30 - Report by Andrew Wood
Affected : Certs & NGE

15:40 - Started review of the CA's
16:30 - Decided to Restart all CA's @US2

17:30 - Issue resolved

Root Cause:

Downstream of PROD-449: Upgrade of Vmware Tools on Us2 Servers. Vendor updated VMWARE tools on servers but since installer didn't request reboot for CA's, they were not rebooted.

As result, CA's behavior was unstable.

Resolution:

After restarting all the CA's at US2, one by one, the service was restored.

Included in procedure used by provider, to reboot the servers after vmware tools upgrade, as a best practice recommended by VMWARE.