On Wednesday, July 10th at approximately 1:10PM UTC, our monitoring infrastructure detected potential SMS issues in our West Coast data center. Investigations determined the cause to be related to internal database complications. The issue was resolved at approximately 1:50PM UTC.
A database sync error was discovered to be caused by an internal back-office software release. As changes were still being made to both databases, overtime this sync error led to data integrity issues, which prevented some SMS transactions from being carried out.
The immediate problem was resolved by manually copying over staged records in the sync to the destination database and re-establishing a new sync connection between the two databases.
The following measures are being taken to prevent an incident of this type from happening in the future:
" Improved reporting, alerting, and response around database sync issues
" Additional database constraints added for mitigation