So, it is 02:26 and my phone goes mental with texts from nagios in Amsterdam. I get up, check laptop and *I* have no connectivity!
We have nagios monitor a lot, from the office, and then we have another nagios in Amsterdam to monitor the monitoring. These texts tell me everything at the office is down...
Fortunately, using my 3G on the iPad I can see that the rest of the world is working (yes, all of it... google was there as well as our broadband lines).
So I am thinking power outage at office? But we have a UPS on the fibre, switch and router. How is it I cannot even see the office from home? That can't be it then.
So I am thinking fibre break. But that would mean both fibres (one to office and one to my house). That is possible maybe, but the DSL backup at the office should kick in - we test it every month!
So I am thinking I have no idea what is going on - cycle to office at 2:30 in the morning. Not funny.
Guess what. The damn UPS has died, so no fibre, no DSL backup, no switch and no fibre links. Arrrrrrg!
Now, this was just our office, but people have asked, on the rare occasions that there are power issues in the data centre, why we don't have UPSs ourselves. The answer is that (a) that almost certainly means a container of acid in a data centre; (b) it means our rack is live when power is off for safety reasons such as a fire; (c) it is probably against DC rules (see a and b); (d) people we connect to would be off so not a big help; (e) the DC has good UPSs; (f) it adds a new point of failure... So if ever we did have UPS in data centre (and were allowed to) we would only have it on one side of the dual feed kit. Today proves how having a UPS can make things less reliable than not having one.
Anyway, back to bed...