Difference between revisions of "Oldwiki.scinet.utoronto.ca:System Alerts"
Line 13: | Line 13: | ||
[[File:up.png|up|link=BGQ]]BGQ | [[File:up.png|up|link=BGQ]]BGQ | ||
[[File:down.png|scratch file system down|link=HPSS]]HPSS | [[File:down.png|scratch file system down|link=HPSS]]HPSS | ||
+ | |||
+ | Sat Aug 10 00:28:54 - Vendor has escalated to yet a higher level of support but still no solution. People will remain on-site for a while longer to see what the new support team recommends. | ||
Fri Aug 9 22:03:48 - Staff and vendor technician remain on-site. Storage vendor has escalated problem to critical but suggested fixes have not yet resolved the problem. BGQ remains up because it has separate filesystem. | Fri Aug 9 22:03:48 - Staff and vendor technician remain on-site. Storage vendor has escalated problem to critical but suggested fixes have not yet resolved the problem. BGQ remains up because it has separate filesystem. |
Revision as of 00:33, 10 August 2013
System Status
Sat Aug 10 00:28:54 - Vendor has escalated to yet a higher level of support but still no solution. People will remain on-site for a while longer to see what the new support team recommends.
Fri Aug 9 22:03:48 - Staff and vendor technician remain on-site. Storage vendor has escalated problem to critical but suggested fixes have not yet resolved the problem. BGQ remains up because it has separate filesystem.
Fri Aug 9 15 32 - /scratch and /project are down. Login and home directories are ok, but no jobs can run, and most of those running will likely die if/when they need to do I/O.
Fri Aug 9 15:25 - File system problems. Scratch is unmounted. Jobs are likely dying. We are working on it.
Thu Aug 8 13:22 - most systems are back up
Thu Aug 8 11:18:45 - problems with storage hardware. Trying to resolve with vendor
Thu Aug 8 08:14:01 Cooling has been restored. Starting to recover systems.
Large voltage drop at site knocked-out cooling system at 0558 today. Staff enroute to site.
Last update: Thu 8 Aug 2013 06:12:28 EDT