Wednesday, March 2, 2011

Outage debrief 2011-03-02

All systems are now operating normally.

At approximately 8:25 AM, our SAN encountered a "Fibre Subsystem Link Failure" which caused all volumes to go offline. Some users at all schools reporting inability to log in, applications quitting unexpectedly, applications not available, Internet not accessible, and general disruption.

Why this did not occur for all users probably has to do with the servers to which each user was authenticated at the time, and the stored location of the file(s) in use.

The cause of the Fibre Subsystem Link Failure was likely a power fluctuation or power failure. However, this should not occur since it has dual redundant power supplies connected to separate battery backup power supply units.

A service company was contacted immediately, and is researching the problem to identify measures we can take to prevent it from happening again.

We apologize for the disruption this caused to our core mission of educating students. Thank you for your patience and flexibility.

No comments: