Follow

Service Interruption – Monday, March 31st, 2014

ArbiterSports experienced a service interruption on Monday March 31st, 2014.  The period of disruption started at 12:59 am ET (9:59 pm PT) and lasted until 2:21 am ET (11:21 pm PT). 

Cause

We experienced a hardware failure on one of our database servers. The error was related to our scheduled redundant backup routine (not our primary backup). We temporarily disabled this service and are working on a fix to avoid this going forward. After disabling the service, we validated the system was online for all clients and that no data was lost.

Post Mortem

We are researching our backup service and putting a patch in place to account for possible errors.

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

Comments

  • Avatar
    Nathan Evans

    During the time that the service was down, system generated emails were not sent. However, when the system was restored the processes were kicked off and sent out the unprocessed emails.

    No data occurred since the primary database was unaffected by the backup process.