[DR TEST] iLab Unavailability Wednesday, December 21st, 2016 23:18:00 EST

[Thursday, December 22, 2016 05:15 UTC] iLab DR team has completed initial verification checks and the web server is available for general use again. A system-wide message is being displayed to all users when they login. This will be up for 4 days. A Reason For Outage report will be published upon completion of the investigation as to the cause of the incident and any relevant next steps.

[Thursday, December 22, 2016 05:05 UTC] iLab DR team has completed the restore of the last full database backup, running system checks now.

[Thursday, December 22, 2016 04:35 UTC] iLab DR team is restoring last full database backup taken at Wednesday, December 21, 2016 at 06:00 UTC. Impact to users will be any data modified/changed or updated since last full database backup cannot be restored. No uploaded files are effected.

[Thursday, December 22, 2016 04:27 UTC] iLab DR team determined that all three database servers are irrecoverable and started the Full Database Recovery Procedure.

[Thursday, December 22, 2016 04:18 UTC] Monitoring services alerted iLab infrastructure and data center teams of a catastrophic failure of the primary database server, primary database slave server, and secondary database slave server. iLab Emergency Communication (EC) team was convened to assess impact and next steps