Thursday, 17 March 2011

Magic smoke and well trained staff!

Well, from 10:07:32 to 10:39:58 today we had no power in Blue Square, Maidenhead.

It looks like a few bits of kit fried too - so looking surge and power cut.

Best I have heard so far is a major UPS went bang, and the smoke tripped an alarm and power safety shut-off and evacuation.

I would just like to say that I am very pleased with how my staff have handled this today - tackling the issues in a sensible priority and updating status pages. As it happened I was running training today, and was able to continue while staff did all the necessary work to resolve matters. It is nice to have a good team.

Broadband was not directly affected and this time the backup systems for DNS and RADIUS worked exactly as expected. Traffic levels showed no glitches at all. The most urgent impact was on VoIP services - sadly our grand designs for more diverse systems fell foul of the practical implementation of many SIP systems, though there may we ways to address this in future. Sadly our Ethernet customers were also affected which is never good for a premium service like this.

Needless to say I'll be chasing for a proper explanation for the data centre. The choice of one that is 10 minutes drive away proved it self today though - we know issues can happen wherever you are, but having staff jump in a car and get it sorted quickly is invaluable.

Well done all.

No comments:

Post a Comment