Weather Disaster Impacts Cyberinfrastructure
IU Bloomington has faced a series of severe weather events that started June 4, and extended to the 15th. The repeated severe weather events have resulted in a series of failures in the electrical power infrastructure supporting Bloomington generally, including IU's advanced computing systems. The events were so severe that Bloomington and surrounding counties have been declared a federal disaster area.
IU's major research computing systems — Big Red, Quarry, and the Data Capacitor — were out of service after the initial power outages on June 4. The period from June 4th to June 15th saw a repeated cycle of putting systems back in production, facing additional weather-related power events, and starting over again. The power event on the 14th of June caused the most equipment loss since the severe weather started.
IU rented emergency generators in order to assure reliable power to restart these systems. These arrived on site Monday, June 16th. The Data Capacitor Lustre-WAN system was online Tuesday, June 17th. The Data Capacitor Lustre-WAN system was used as parallel I/O filesystem for Big Red, Quarry and Libra while we completed recovery of the GPFS system.
Quarry returned to service Wednesday, June 18th. Big Red was in full production Thursday, June 19. GPFS became available on Friday, June 20. On Tuesday, June 24th, the Data Capacitor was mounted on Big Red and LEAD project filespace made available. The Data Capacitor will be restored completely during scheduled maintenance on Tuesday, July 8.
We recognize the impact that this downtime has on researchers within IU and nationally. Our teams are working as hard as we can, with all the resources we can bring to bear to get IU research systems running again as quickly as possible.
If you face deadlines in the next few days, please contact us at researchtechnologies@iu.edu and we will work with you to find alternate solutions for your immediate computing needs.
IU's new massive Data Center, currently under construction, will be a hardened facility. It is designed in such a way that it will not be subject to even the extraordinary series of weather events that brought about the current situation.
Photo: aerial view of construction site, May 13th
